Backup over WAN without VPN?

[Jason2312312]

I have two PVE hosts at Site A with a 1 Gbit fiber uplink. Site B (900 Mbit download / 90 Mbit upload) is connected over an IPsec tunnel with a virtual PBS server that handles the backups. Currently, the maximum throughput is 200 Mbit because the VPN encryption is too heavy on the routers. I confirmed this with Iperf3 and observed that the routers' CPUs are at 100% when the throughput reaches around 230 Mbit.

Although 20 MB per second is not bad for a remote backup, it could be about 4 times better. I need to back up around 5 TB of data, which currently takes about 3 days. Since the bitmap sometimes gets cleared, I am unable to run daily backups consistently. My goal is to run backups every 4 hours.

I am considering backing up directly over WAN and want to understand the implications of this approach. Specifically, I am unsure if the connection will be secure enough, I plan to whitelist the WAN IP of Site A.

 

[VictorSTS]

Since the bitmap sometimes gets cleared, I am unable to run daily backups consistently

If dirty map is lost for any reason, the PVE host has to read the whole VM disk but it sends the chunk to PBS only if it doesn't exist there. That is, losing dirity map doesn't generate "a lot" more traffic than a backup with dirty map.

I am considering backing up directly over WAN and want to understand the implications of this approach.

All traffic between PVE and PBS is encrypted with TLS, it is as secure as TLS is.

 

[Jason2312312]

Sounds good, since my Windows VMs all are Bitlocker encrypted I choose not to further encrypt the backups, I just set it up with IP whitelist and the first results are very promising:

OLD speeds


New speeds:


Very happy with this, if anyone still has anything to add about the security of doing this over WAN please tell.

 

[sw-omit]

If your routers can't handle the encryption speeds (and you don't want to / can't replace them for ones that either can support it, or have/can use WireGuard and/or OpenVPN), one other option would be to move the tunnel-creation off-router, for example into a OpnSense-VM and make a static route for the network on the other side on both PVE and PBS to use this router to reach the other Site.

Just throwing ideas out there, but good to see/hear that it works for you (and of course whatever method used, this direct-connect / port-forward method will not be beaten in speeds of course.

 

[VictorSTS]

Forgot to suggets you to read about how QEMU backup works and use fleecing [1] as a good method to avoid problems with PBS disconnecting in the middle of a backup.

[1] https://pve.proxmox.com/pve-docs/chapter-vzdump.html#_vm_backup_fleecing

 

[FormerVMW]

Forgot to suggets you to read about how QEMU backup works and use fleecing [1] as a good method to avoid problems with PBS disconnecting in the middle of a backup.

[1] https://pve.proxmox.com/pve-docs/chapter-vzdump.html#_vm_backup_fleecing

Anyone not using fleecing should enable it. Since it's addition to the backup options, there are likely a lot of people who have existing backup schedules setup and are not using it.

 

[Jason2312312]

If your routers can't handle the encryption speeds (and you don't want to / can't replace them for ones that either can support it, or have/can use WireGuard and/or OpenVPN), one other option would be to move the tunnel-creation off-router, for example into a OpnSense-VM and make a static route for the network on the other side on both PVE and PBS to use this router to reach the other Site.

Just throwing ideas out there, but good to see/hear that it works for you (and of course whatever method used, this direct-connect / port-forward method will not be beaten in speeds of course.

Good idea, hadn't thought of using a VM to build the tunnel on both sides, I do think that it would add complexity to my setup which I want to avoid, it would also mean I am reliant on the VM running in case of having to restore the backup, which would be impossible if the host dies. For now I think the direct connection is my best option, until I get better hardware to build the tunnel, probably a Netgate firewall will be my choice.

The suggestion for fleecing is appreciated, I will look into it next week.

Thanks!

 

[noire]

Hey there, Jason, i am interested into this as well could u give a quick brief about how you did accomplish this?
Many thanks

 

[Jason2312312]

Hey there, Jason, i am interested into this as well could u give a quick brief about how you did accomplish this?
Many thanks

Hi Noire,
Is there a specific part that you can't accomplish? I just added the PBS to the host via public IP, so you need to set up port forwarding and ip whitelisting for extra security.

My speeds are now around 80MB/s which is great.