[Feature Request] Simultaneous connections while creating and restoring backups

[aeminkocal]

Hello,
I have 1Gbps connection at my home, yet the backup speed goes down up to 8 mbps highly because of my ISP's intercontinental connection protocols or settings.
When I create local tar.gz dumps, then download them simultaneously using aria2c with 16 connections per file, I am able to get 800Mbps from the same server to my home server.
The same method is used to make downloads and uploads faster on many other applications. IDM could be a instance.
I have contacted to my ISP multiple times yes they said they are unable to fix the issue.

If we could create multiple connections (maybe up to 128 connections per backup job), it could increase speed for those who are in my state.

 

[fabian]

we don't want to do multiple connections (we do multiple http2 streams over a single connection) - else the number of connections at the server side would explode.

 

[aeminkocal]

we don't want to do multiple connections (we do multiple http2 streams over a single connection) - else the number of connections at the server side would explode.

Can I increase http2 stream count somehow? Is seems to be not enough in my case.

 

[fabian]

no, it's not configurable (http2 will adapt the number of concurrent streams dynamically).

 

[aeminkocal]

This is when I download files simultaneously with 16 connections per file from ftp using aria2c using "-s16 -x16" arguments:

And this is the result when I try to use proxmox backup server:

If I cannot change anything, then do you have any other suggestions?

 

[fabian]

if something/someone on your network is rate-limiting long-running connections, that is something you have to take up with your network admins/network provider..

 

[lucius_the]

Would like to add to this feature request...
I have PBS on an older server, that only has 1GBE adapters. So I used LACP from server to switch to add bandwidth. However, if PVE uses only one connection to PBS for backup, then I can't leverage LACP/LAG. If there was, at least an option, to configure multiple TCP streams from PVE to PBS during backup (andl during restore) that would double my rate...

 

[fabian]

somewhat related: https://bugzilla.proxmox.com/show_bug.cgi?id=3347

 

[lucius_the]

somewhat related: https://bugzilla.proxmox.com/show_bug.cgi?id=3347

Yes...
But I do have layer3+4 on the switch. And it seems to work when I try with iperf between PVE and PBS.

In any case, do think about it. Most backup systems use multiple streams for various reasons:
- better bandwidth allocation on Internet or slower WAN links
- better multicore utilization
- etc
Would be good to have it.

In any case, just switched my company from ESXi to Promox -> most things work beautifully. THANK YOU!