Remote sync very slow (0,4 MiB/s)

[carsten2]

The remote sync is very slow. It transfers only 0.4 MiB/s and takes 1h20min to transfer 2GB. See the log.

2020-11-15T09:50:18+01:00: sync snapshot "vm/101/2020-10-13T22:00:02Z"
2020-11-15T09:50:19+01:00: sync archive qemu-server.conf.blob
2020-11-15T09:50:19+01:00: sync archive drive-virtio0.img.fidx
2020-11-15T11:00:30+01:00: downloaded 1775574586 bytes (0.40 MiB/s)
2020-11-15T11:00:30+01:00: sync archive drive-virtio1.img.fidx
2020-11-15T11:12:32+01:00: downloaded 298909805 bytes (0.41 MiB/s)
2020-11-15T11:12:32+01:00: got backup log file "client.log.blob"
2020-11-15T11:12:32+01:00: sync snapshot "vm/101/2020-10-13T22:00:02Z" done

Also because the sync is that slow, it produces timeouts like the follwing

2020-11-15T09:20:04+01:00: sync group vm/101 failed - authentication failed - invalid ticket - expired

BTW: Please change numbers to human readable numbers as suggested in a bugzilla ticket. 298909805 / 1775574586 is much less readable than 298 MiB / 1775 MiB.

 

[t.lamprecht]

The remote sync is very slow. It transfers only 0.4 MiB/s and takes 1h20min to transfer 2GB. See the log.

Can you please describe the network between the two instances?
* Are you tunneling it through some layer? If yes, can you try without tunneling (or another more lightweight one) to see if that is the producing some of that effects.
* what are latencies like between those?
* what is the client benchmark output between those

Also because the sync is that slow, it produces timeouts like the follwing

What versions are you on, as on 1.0 that should not happen (renewing the auth/ticket runs in an independent, periodic task there).

 

[carsten2]

benchmark from remote site to local site:

local network
Uploaded 143 chunks in 5 seconds.
Time per request: 36770 microseconds.
TLS speed: 114.07 MB/s
SHA256 speed: 450.80 MB/s
Compression speed: 591.60 MB/s
Decompress speed: 1029.48 MB/s
AES256/GCM speed: 2179.01 MB/s
Verify speed: 317.33 MB/s

Remote to local over sshuttle (ssh-tunnel)
Uploaded 10 chunks in 182 seconds.
Time per request: 18227215 microseconds.
TLS speed: 0.23 MB/s
SHA256 speed: 460.78 MB/s
Compression speed: 602.34 MB/s
Decompress speed: 1039.39 MB/s
AES256/GCM speed: 2176.49 MB/s
Verify speed: 318.10 MB/s

Remote to local directly over port 8007
Uploaded 10 chunks in 119 seconds.
Time per request: 11966427 microseconds.
TLS speed: 0.35 MB/s
SHA256 speed: 461.34 MB/s
Compression speed: 598.53 MB/s
Decompress speed: 1032.67 MB/s
AES256/GCM speed: 2180.20 MB/s
Verify speed: 319.37 MB/s

How to test TCP-latency?

I am running 1.0 (but don't know exactly, when the update took place). I will see whether the error appears again.

 

[Cookiefamily]

Sshuttle (while being awesome) is notoriously slow. You might consider using a „real“ VPN like wireguard, those should achieve more throughout.

 

[carsten2]

Sshuttle (while being awesome) is notoriously slow. You might consider using a „real“ VPN like wireguard, those should achieve more throughout.

It is not sshuttle because the direct connection via port 8007 (even no VPN) is as slow as the sshuttle connection.

 

[t.lamprecht]

How to test TCP-latency?

ping would help as a basic starter.

It is not sshuttle because the direct connection via port 8007 (even no VPN) is as slow as the sshuttle connection.

I mean the non-tunneled connection needs about a minute less time to complete, so there's that...

 

[carsten2]

Ping is 50ms (offsite over internet)
Yes, direct connection is faster, but still incredibly slow.

BTW: The reasons I use SSH is:
a) I dont't want to expose the GUI and the fact, that there is a PBS running. Also a complete UI-interface offers more attack surface.
b) With SSH I get fail2ban and other firewall attach filtering, which seems to be missing with PBS direct connection. At least fail2ban should be supported out of the box by PBS.

 

[t.lamprecht]

Ping is 50ms (offsite over internet)
Yes, direct connection is faster, but still incredibly slow.

What is your upload bandwidth on the source, and the download bandwidth on the target?
Or, what does an iperf test says about the maximal practical bandwidth available?

a) I dont't want to expose the GUI and the fact, that there is a PBS running. Also a complete UI-interface offers more attack surface.

No, not really, it's always just using the API and so it's reduced to the attack surface of the API.
It does not help for security at all, but it's your setup after all, just noting

b) With SSH I get fail2ban and other firewall attach filtering, which seems to be missing with PBS direct connection. At least fail2ban should be supported out of the box by PBS.

You can already setup fail2ban on /var/log/proxmox-backup/api/auth.log, it is formatted such that the PVE docs should only need a few lightweight changes: https://pve.proxmox.com/wiki/Fail2ban

 

[carsten2]

You can already setup fail2ban on /var/log/proxmox-backup/api/auth.log, it is formatted such that the PVE docs should only need a few lightweight changes: https://pve.proxmox.com/wiki/Fail2ban

PBS should come with preconfigured fail2ban support. No service should be exposed to the internet without it. This is one reason I choose ssh tunnel, because everything with security works out of the box.

 

[dbayer]

Has anything been figured out on this issue?

I'm dealing with a similar situation. I'm directly connected with no tunneling and getting around .80MiB/s. The slowest link in this connection is the download speed at the remote site, which is a theoretical 5MiB/s

Thanks,
Daniel

 

[dbayer]

Bump.

 

[dcsapak]

Bump.

please do not bump threads without any additional information

I'm dealing with a similar situation. I'm directly connected with no tunneling and getting around .80MiB/s. The slowest link in this connection is the download speed at the remote site, which is a theoretical 5MiB/s

what is the latency in between? can you test the bandwidth with iperf?

 

[tommytom]

i have also a very slow sync speed running Backup Server 1.0-6 on both ends.
network connection 1000mb down 50mb up

no matter if i use the connection over wireguard (as shown below) or over the direct network. The speed is very slow regardless of the connection.
very confusing for me was, that it sometimes download at full speed with about 40mb/s
and then all over sudden it dropped to maybe 1mb/s

when i test wget and download a file from the same destination i reach 40mb/s +

hope that helps to investigate this

root@pbs:~# ping 192.168.159.228 PING 192.168.159.228 (192.168.159.228) 56(84) bytes of data. 64 bytes from 192.168.159.228: icmp_seq=1 ttl=62 time=27.5 ms 64 bytes from 192.168.159.228: icmp_seq=2 ttl=62 time=41.3 ms 64 bytes from 192.168.159.228: icmp_seq=3 ttl=62 time=33.0 ms 64 bytes from 192.168.159.228: icmp_seq=4 ttl=62 time=31.6 ms 64 bytes from 192.168.159.228: icmp_seq=5 ttl=62 time=28.8 ms 64 bytes from 192.168.159.228: icmp_seq=6 ttl=62 time=25.8 ms

root@pbs:~# proxmox-backup-client benchmark --repository 192.168.159.228:Backup
Uploaded 13 chunks in 16 seconds.
Time per request: 1300976 microseconds.
TLS speed: 3.22 MB/s
SHA256 speed: 427.05 MB/s
Compression speed: 653.46 MB/s
Decompress speed: 1113.60 MB/s
AES256/GCM speed: 2441.89 MB/s
Verify speed: 300.89 MB/s
┌───────────────────────────────────┬────────────────────┐
│ Name                              │ Value              │
╞═══════════════════════════════════╪════════════════════╡
│ TLS (maximal backup upload speed) │ 3.22 MB/s (0%)     │
├───────────────────────────────────┼────────────────────┤
│ SHA256 checksum computation speed │ 427.05 MB/s (21%)  │
├───────────────────────────────────┼────────────────────┤
│ ZStd level 1 compression speed    │ 653.46 MB/s (87%)  │
├───────────────────────────────────┼────────────────────┤
│ ZStd level 1 decompression speed  │ 1113.60 MB/s (93%) │
├───────────────────────────────────┼────────────────────┤
│ Chunk verification speed          │ 300.89 MB/s (40%)  │
├───────────────────────────────────┼────────────────────┤
│ AES256 GCM encryption speed       │ 2441.89 MB/s (67%) │
└───────────────────────────────────┴────────────────────┘

ProxmoxBackup Server 1.0-6
CPU
Memory
Root Disk
No valid subscription
You do not have a valid subscription for this server. Please visit www.proxmox.com to get a list of available options.
()
2021-02-08T16:46:56+01:00: starting new backup reader datastore 'Backup': "/mnt/Backup/"
2021-02-08T16:46:56+01:00: protocol upgrade done
2021-02-08T16:46:56+01:00: GET /download
2021-02-08T16:46:56+01:00: download "/mnt/Backup/vm/242/2021-02-07T18:18:25Z/index.json.blob"
2021-02-08T16:46:56+01:00: GET /download
2021-02-08T16:46:56+01:00: download "/mnt/Backup/vm/242/2021-02-07T18:18:25Z/drive-virtio1.img.fidx"
2021-02-08T16:46:56+01:00: register chunks in 'drive-virtio1.img.fidx' as downloadable.
2021-02-08T16:47:10+01:00: GET /chunk
2021-02-08T16:47:10+01:00: download chunk "/mnt/Backup/.chunks/4d15/4d15dd406b5cc4f803aef79759089e787e3353a5283f69adce6296e4e7bfaf22"
2021-02-08T16:47:10+01:00: GET /chunk
2021-02-08T16:47:10+01:00: GET /chunk
2021-02-08T16:47:10+01:00: GET /chunk
2021-02-08T16:47:10+01:00: GET /chunk
2021-02-08T16:47:10+01:00: GET /chunk
2021-02-08T16:47:10+01:00: download chunk "/mnt/Backup/.chunks/c2c6/c2c60f3858faf207f9dfd07ad63f18b788f0a51f356330c43c01b9d1cb0f8ec8"
2021-02-08T16:47:10+01:00: download chunk "/mnt/Backup/.chunks/9815/981576fac0d4c7b96e44360a78fcbb12e152ae73cdbce3c2d60e1470812ba6fb"
2021-02-08T16:47:10+01:00: download chunk "/mnt/Backup/.chunks/9a8f/9a8f63a8f7fbb97b80b36da1fdf37bcb4814a799b6a4dc75a0e42fd982e0213b"
2021-02-08T16:47:10+01:00: download chunk "/mnt/Backup/.chunks/7eaa/7eaa1a3c3193cd9e85ae91870e45541949610b0ab1703383f0e2f354a230d7f6"
2021-02-08T16:47:10+01:00: GET /chunk
2021-02-08T16:47:10+01:00: GET /chunk
2021-02-08T16:47:10+01:00: GET /chunk
2021-02-08T16:47:10+01:00: GET /chunk
2021-02-08T16:47:10+01:00: download chunk "/mnt/Backup/.chunks/f7ab/f7abdafa6aed0bd7e21c331bc5fc5f36f72620eb148387a161294dd998fc005e"
2021-02-08T16:47:10+01:00: download chunk "/mnt/Backup/.chunks/1983/19836318f89c2ca299bda016d36b268052fe5540387511ebcd0431d20254b0f5"
2021-02-08T16:47:10+01:00: download chunk "/mnt/Backup/.chunks/47bc/47bcb02c0da3df09b393b54c2eaf5015af32c91386ff80ab7a8c469054798b9e"
2021-02-08T16:47:10+01:00: download chunk "/mnt/Backup/.chunks/5ea3/5ea32a161291e1b8e7eaf1be454d27bda7851844541d5474d639f5507ded97a2"
2021-02-08T16:47:10+01:00: GET /chunk
2021-02-08T16:47:10+01:00: GET /chunk
2021-02-08T16:47:10+01:00: GET /chunk
2021-02-08T16:47:10+01:00: GET /chunk
2021-02-08T16:47:10+01:00: GET /chunk
2021-02-08T16:47:10+01:00: download chunk "/mnt/Backup/.chunks/ae05/ae05a951edbd1efc62592eb6cc6b968498c47ba2b050a7801e5ebb174b039c1c"
2021-02-08T16:47:10+01:00: download chunk "/mnt/Backup/.chunks/edc0/edc042a6c13067d3f29c6961187029bb37562f1da696816560a00fef900a8aa3"
2021-02-08T16:47:10+01:00: GET /chunk
2021-02-08T16:47:10+01:00: download chunk "/mnt/Backup/.chunks/49ef/49ef10c715bd0d1e009f57025aff5b36fe5debea3f99825a4149e7a9b2bb1fef"
2021-02-08T16:47:10+01:00: download chunk "/mnt/Backup/.chunks/91d8/91d8126896f38244ed36ee161fa898e7243be83f5832e2349dbac146c0dd04e7"
2021-02-08T16:47:10+01:00: download chunk "/mnt/Backup/.chunks/9782/978244081548e75f7c327fd03e3cc245da812b34062fcf2e16e7042ee93625f8"
2021-02-08T16:47:10+01:00: download chunk "/mnt/Backup/.chunks/2b24/2b24bbbfcca9508b9fa8bfd26da2adee6e2544ec479eb0100b427e9c856fc62f"
2021-02-08T16:47:10+01:00: GET /chunk
2021-02-08T16:47:10+01:00: GET /chunk
2021-02-08T16:47:10+01:00: download chunk "/mnt/Backup/.chunks/6c71/6c71fcb77ddd9e4e47cfd6b986d6d6ed37ab80b16e5c7e9c8a564779ede63694"
2021-02-08T16:47:10+01:00: GET /chunk
2021-02-08T16:47:10+01:00: download chunk "/mnt/Backup/.chunks/3b1e/3b1e2e227b78cd858757cf2413ba1e1fb56de10307df9c2f028bca9e9780e914"
2021-02-08T16:47:10+01:00: GET /chunk
2021-02-08T16:47:10+01:00: download chunk "/mnt/Backup/.chunks/0df3/0df3c36aaadd2c1dff919667dd2513ba6350814b24a611898ae27ae43a007090"
2021-02-08T16:47:10+01:00: download chunk "/mnt/Backup/.chunks/3b2f/3b2fbf6963834115c1555fea450c0ee58f26e40b508cf1c6bc69897a135a67bd"
2021-02-08T16:47:10+01:00: download chunk "/mnt/Backup/.chunks/d845/d84585c515a3ee29894c615254629b89569c2ae2457aa13c56296499971c6817"
2021-02-08T16:47:11+01:00: GET /chunk
2021-02-08T16:47:11+01:00: download chunk "/mnt/Backup/.chunks/5479/547904dccb456bb5eedba8c6692af7337c815c49c25fc7ea6d0dfbd6a36d8c51"
2021-02-08T16:47:12+01:00: GET /chunk
2021-02-08T16:47:12+01:00: download chunk "/mnt/Backup/.chunks/e7c6/e7c60accd2837c6d5e8645e5fdb3f41fe1fad3a2af74c79967b949bd05a60a26"
2021-02-08T16:47:13+01:00: GET /chunk
2021-02-08T16:47:13+01:00: download chunk "/mnt/Backup/.chunks/5bc5/5bc5fe8c28745145908f2e5af277377a2bfd970881698d27921362dc088b3511"
2021-02-08T16:47:14+01:00: GET /chunk
2021-02-08T16:47:14+01:00: download chunk "/mnt/Backup/.chunks/939c/939c8463f1fadfeb9f6dc6269ecbaac07bee61ffbc65ce0d490a4af9ccac4801"
2021-02-08T16:47:15+01:00: GET /chunk
2021-02-08T16:47:15+01:00: download chunk "/mnt/Backup/.chunks/3a1b/3a1b60198eea7fe645737d3f2d1c3fe2aef508ea5aea736aa9b7ab494620bd44"
2021-02-08T16:47:15+01:00: GET /chunk
2021-02-08T16:47:15+01:00: download chunk "/mnt/Backup/.chunks/5d47/5d479910b31d73f53956a863f5efc86132a4a18a6b82bdc6e3654fc09ea5fc8c"
2021-02-08T16:47:16+01:00: GET /chunk
2021-02-08T16:47:16+01:00: download chunk "/mnt/Backup/.chunks/d54f/d54f476378f7eed1914885d60f88c3e468d1ddb5465afb001edab583677f17fa"
2021-02-08T16:47:16+01:00: GET /chunk
2021-02-08T16:47:16+01:00: download chunk "/mnt/Backup/.chunks/862d/862de2a338cc2fbdeee376b123a8f7e47548ec392e7b642403af347898562153"
2021-02-08T16:47:17+01:00: GET /chunk
2021-02-08T16:47:17+01:00: download chunk "/mnt/Backup/.chunks/684a/684ab9b68ac3c0d1d9eae344ae7614fceca754297a29bc5318536ab8fa602d9a"
2021-02-08T16:47:17+01:00: GET /chunk
2021-02-08T16:47:17+01:00: download chunk "/mnt/Backup/.chunks/2a48/2a4812921946167b1d1fa4200b71d45701fa92292c1c168e213ea6e74bad5581"
2021-02-08T16:47:18+01:00: GET /chunk
2021-02-08T16:47:18+01:00: download chunk "/mnt/Backup/.chunks/d9b7/d9b7d75b74f48f4ac77f5598f1ebeb8aa855752eabba1c692ad6f274b388856f"
2021-02-08T16:47:18+01:00: GET /chunk
2021-02-08T16:47:18+01:00: download chunk "/mnt/Backup/.chunks/9e40/9e40d589fb98f02d186f2c7dc4d52bb0bf3b29514dbf06c8110d85d0dd31f7f1"
2021-02-08T16:47:19+01:00: GET /chunk
2021-02-08T16:47:19+01:00: download chunk "/mnt/Backup/.chunks/d4b4/d4b4a5df1e96de176e9fbba507827d4f4885395002017bc3bec1b3d5fe1977e6"
2021-02-08T16:47:19+01:00: GET /chunk
2021-02-08T16:47:19+01:00: download chunk "/mnt/Backup/.chunks/9aa5/9aa57e79f966095acb429788eecd54f027199060db70dd0ff6e6874f49486740"
2021-02-08T16:47:19+01:00: GET /chunk
2021-02-08T16:47:19+01:00: download chunk "/mnt/Backup/.chunks/8c31/8c316f9b2f3471985c17324f815bd4653270b1666af256aadc7c27cbaae03207"
2021-02-08T16:47:20+01:00: GET /chunk
2021-02-08T16:47:20+01:00: download chunk "/mnt/Backup/.chunks/1830/1830f83a44bfa3cfacfbfe0daac79d27e91063e6b738c8984ee5560e16f9b84b"
2021-02-08T16:47:20+01:00: GET /chunk
2021-02-08T16:47:20+01:00: download chunk "/mnt/Backup/.chunks/928a/928a6b8068ca3668d80692f56720e150d7cb0178923c51b5606cc7c604343aa8"
2021-02-08T16:47:21+01:00: GET /chunk
2021-02-08T16:47:21+01:00: download chunk "/mnt/Backup/.chunks/2afd/2afdbca1ec266e32e8ec36fe1da712e84d5a78e74eafff53ac107c14f430b021"
2021-02-08T16:47:21+01:00: GET /chunk
2021-02-08T16:47:21+01:00: download chunk "/mnt/Backup/.chunks/4d5a/4d5aa6a19ae72416f479729f01eb913f44986265b117b14dd3e5cf3068e2a050"
2021-02-08T16:47:21+01:00: GET /chunk
2021-02-08T16:47:21+01:00: download chunk "/mnt/Backup/.chunks/8f04/8f047b0f191ae2d6dc4a6838f3b4504dafdfb165a7af31e527b8f48ff4d8e9e9"
2021-02-08T16:47:22+01:00: GET /chunk
2021-02-08T16:47:22+01:00: download chunk "/mnt/Backup/.chunks/47f5/47f5060c2655809f38f7e4b3fb3c5064071d7a3aa6e335efbf5ea32364a554f0"
2021-02-08T16:47:22+01:00: GET /chunk
2021-02-08T16:47:22+01:00: download chunk "/mnt/Backup/.chunks/883e/883ec8843ab6c1fd62ef1a20f6485f362ac9d9faa61476f1db2371c4c6b76481"
2021-02-08T16:47:23+01:00: GET /chunk
2021-02-08T16:47:23+01:00: download chunk "/mnt/Backup/.chunks/4dcd/4dcdf5559be44289940261ff5152eae315a0ca8b8577f1ecfaa29f22fecdbee8"
2021-02-08T16:47:23+01:00: GET /chunk
2021-02-08T16:47:23+01:00: download chunk "/mnt/Backup/.chunks/47cb/47cb1a729ac5e0643e0be4d3b7fa3a5f72ef98c05a081ae75433dc38db37005f"
2021-02-08T16:47:24+01:00: GET /chunk
2021-02-08T16:47:24+01:00: download chunk "/mnt/Backup/.chunks/5491/54914143ddf5ded8ad23cb1d3692209fc5c79edb999015f6d5831f3e12833094"
2021-02-08T16:47:24+01:00: GET /chunk
2021-02-08T16:47:24+01:00: download chunk "/mnt/Backup/.chunks/18ba/18baf6f8a59e70f5ea8c98b24c761d263a491032202374f8bfd9be919436a975"
2021-02-08T16:47:25+01:00: GET /chunk
2021-02-08T16:47:25+01:00: download chunk "/mnt/Backup/.chunks/de92/de92d11590cafb3532b162acf5723c087bf851c6cd9a575c1a0a69ee3166530d"
2021-02-08T16:47:25+01:00: GET /chunk
2021-02-08T16:47:25+01:00: download chunk "/mnt/Backup/.chunks/8c8d/8c8d25bd3dbd291e3d3ba40c01ef5ac03311148e59f99526065f4c45e293bdb5"
2021-02-08T16:47:25+01:00: GET /chunk
2021-02-08T16:47:25+01:00: download chunk "/mnt/Backup/.chunks/45c8/45c8ac41a569151a6a8c87b0be9b7baecb9143a5e7b0e4e96eb59096bdeab1d6"
2021-02-08T16:47:26+01:00: GET /chunk
2021-02-08T16:47:26+01:00: download chunk "/mnt/Backup/.chunks/c295/c295ac13dfb4aee468673bbbe40ba0a7a5d02c313fb9c35e07e5b031491cbb5d"
2021-02-08T16:47:26+01:00: GET /chunk
2021-02-08T16:47:26+01:00: download chunk "/mnt/Backup/.chunks/f530/f5308201fe7492d379d06bd817b7d40f2b097ea154b0d3cdeec3cf71d1e6b563"
2021-02-08T16:47:27+01:00: GET /chunk
2021-02-08T16:47:27+01:00: download chunk "/mnt/Backup/.chunks/5dc7/5dc7382ade94fdd58f70e24cde1b463a6eedd490f35aa702b7ee150693364b17"
2021-02-08T16:47:27+01:00: GET /chunk
2021-02-08T16:47:27+01:00: download chunk "/mnt/Backup/.chunks/8b3c/8b3cb36b5bc9a64142325525d94fc18020582f394cb871d94055a7d67bd9edef"
2021-02-08T16:47:28+01:00: GET /chunk
2021-02-08T16:47:28+01:00: download chunk "/mnt/Backup/.chunks/0200/02007b3c5224fe28cb3fd97234f6ddb0a4a091d50fceca152a9b3317afcc5a40"
2021-02-08T16:47:28+01:00: GET /chunk
2021-02-08T16:47:28+01:00: download chunk "/mnt/Backup/.chunks/64e5/64e544f495f43eee0075bef602f50adf091bacb63e7eed37bd94fac22c0c936f"
2021-02-08T16:47:29+01:00: GET /chunk
2021-02-08T16:47:29+01:00: download chunk "/mnt/Backup/.chunks/fd02/fd02b7c583563c69e27da625a1cfbf8613d8136a0100ce128cb299f60b7596c6"
2021-02-08T16:47:29+01:00: GET /chunk
2021-02-08T16:47:29+01:00: download chunk "/mnt/Backup/.chunks/f6c7/f6c7a86a2d050539098c2de1532156ed3e88c9776484699458af7fbe0f31cb25"
2021-02-08T16:47:29+01:00: GET /chunk
2021-02-08T16:47:29+01:00: download chunk "/mnt/Backup/.chunks/5cc7/5cc745db8e4c4fd799ff6b247cccaab5524900099a57611fc3dbf505f45d90c2"
2021-02-08T16:47:30+01:00: GET /chunk
2021-02-08T16:47:30+01:00: download chunk "/mnt/Backup/.chunks/c2f7/c2f7bbe98531a504194e08f1f684ed67dadb770067d4c4d6d89b05435ae65372"
2021-02-08T16:47:30+01:00: GET /chunk
2021-02-08T16:47:30+01:00: download chunk "/mnt/Backup/.chunks/674d/674d90371a8593f9dd1bd73a682ea399c6fdf0780a3f53755646863dd0c1676c"
2021-02-08T16:47:31+01:00: GET /chunk
2021-02-08T16:47:31+01:00: download chunk "/mnt/Backup/.chunks/874c/874c1be5541cc8b670efbd850da3769d6b72aaa28e3fd4461ca703de62504bff"
2021-02-08T16:47:31+01:00: GET /chunk
2021-02-08T16:47:31+01:00: download chunk "/mnt/Backup/.chunks/af75/af7557b43e1c104460a9ff52d536e4b27a19738e9cebb477e14f1004557ce33d"
2021-02-08T16:47:32+01:00: GET /chunk
2021-02-08T16:47:32+01:00: download chunk "/mnt/Backup/.chunks/c7eb/c7eb0ddcb8f30dc4641b4780bd537f4c37a9d9231bb372195ffd5c002110bac1"
2021-02-08T16:47:32+01:00: GET /chunk
2021-02-08T16:47:32+01:00: download chunk "/mnt/Backup/.chunks/3aab/3aabbb4e6b602f465074d6f23eec215bcca2d116e9d288226b530b8cbe60b46e"

 

[dcsapak]

network connection 1000mb down 50mb up

are you talking mbyte or mbit ? for mbit, the result does not look too bad since 3.22MiB -> 25MBit/s so roughly half

please post an iperf test in both directions and also 'proxmox-backup-client benchmark' in both directions

 

[tommytom]

are you talking mbyte or mbit ? for mbit, the result does not look too bad since 3.22MiB -> 25MBit/s so roughly half

please post an iperf test in both directions and also 'proxmox-backup-client benchmark' in both directions

to clarify: i have 1 gigabit downstream and 50mbit Upstream on my home sync PBS site which produces slow speed.

the main backup-server server is connected with 10gigabit to the network 30-50ms ping time away

as i wrote.. sometimes i had even 40megabyte download speed. before it dropped on a very huge vm with 1.2Tb disk size.

 

[dcsapak]

to clarify: i have 1 gigabit downstream and 50mbit Upstream on my home sync PBS site which produces slow speed.

the main backup-server server is connected with 10gigabit to the network 30-50ms ping time away

it is still not clear in which direction you did benchmark, also does the backup-server really have a 10gbit uplink?

as i wrote.. sometimes i had even 40megabyte download speed. before it dropped on a very huge vm with 1.2Tb disk size.

this also very much depends on the storage + caches, e.g. if you have 'slow' mechanical disks and the chunks are not in the memory cache, speeds will be much lower
this: https://git.proxmox.com/?p=proxmox-...0f58d259606b643df542e2664b08009fe;hb=HEAD#l64
has some napkin math calculations for how much speed you can get, but of course higher latency links will be detrimental to speed

 

[tommytom]

False alarm!
It's because of our network connection. We have a 1GigaBit line with Vodafone as a business customer. Unbelievably, they throttle the line down to 10mbit during the day. Shortly after midnight, everything works at full speed. Then you also get a good performance with the PBS as expected.

I cannot recommend Vodafone in Germany. We will now probably get a line with Colt.

Thanks for your great work !

 

[dbayer]

False alarm!
It's because of our network connection. We have a 1GigaBit line with Vodafone as a business customer. Unbelievably, they throttle the line down to 10mbit during the day. Shortly after midnight, everything works at full speed. Then you also get a good performance with the PBS as expected.

I cannot recommend Vodafone in Germany. We will now probably get a line with Colt.

Thanks for your great work !

How did you discover they were throttling?

 

[tommytom]

because i checked the line frequently and found out, that single line connections like with wget a throttled in the evening hours and get back to fullspeed short after 00:00 am - every day... from up to 50mbyte/s they throttle it to 500kb-800kb/s