[SOLVED] can't get backups to PBS working over pfsense wireguard tunnel

[marc_things]

I have 2 PVE hosts and one PBS host.

PVE A is located at the same location as PBS and works fine.
PVE B is at a second location and backups seem to get stuck at 0% and never progress.

Both sites are running pfsense firewalls.

I have a wireguard site-to-site tunnel between each.

MTU & MSS on both pfSense wireguard interfaces is set to 1420. I've tried 1200 as well with the same results.

I was able to add PBS datastore to PVE B without any issues but when I try to backup it gets to 0% and just hangs there.

My goal is to be able to backup VMs hosted on PVE B to the PBS server over the wireguard site to site tunnel.

INFO: starting new backup job: vzdump 401 --mode snapshot --notes-template '{{guestname}}' --storage backupPool --remove 0 --node beemox --notification-mode auto
INFO: Starting Backup of VM 401 (qemu)
INFO: Backup started at 2025-01-08 23:58:11
INFO: status = running
INFO: VM Name: RA002S
INFO: include disk 'scsi0' 'beepool:vm-401-disk-0' 120G
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating Proxmox Backup Server archive 'vm/401/2025-01-09T04:58:11Z'
INFO: enabling encryption
INFO: issuing guest-agent 'fs-freeze' command
INFO: issuing guest-agent 'fs-thaw' command
INFO: started backup task '0d899831-994f-40e2-b56a-3f0be22c2122'
INFO: resuming VM again
INFO: scsi0: dirty-bitmap status: existing bitmap was invalid and has been cleared
INFO:   0% (212.0 MiB of 120.0 GiB) in 3s, read: 70.7 MiB/s, write: 30.7 MiB/s

here is the info from the output on the Backup task summary on PBS:

2025-01-08T23:58:12-05:00: GET /previous: 400 Bad Request: no valid previous backup
2025-01-08T23:58:12-05:00: created new fixed index 1 ("vm/401/2025-01-09T04:58:11Z/drive-scsi0.img.fidx")
2025-01-08T23:58:12-05:00: add blob "/mnt/datastore/backupPool/vm/401/2025-01-09T04:58:11Z/qemu-server.conf.blob" (404 bytes, comp: 404)
2025-01-09T00:03:29-05:00: POST /fixed_chunk: 400 Bad Request: error reading a body from connection: bytes remaining on stream
2025-01-09T00:03:29-05:00: POST /fixed_chunk: 400 Bad Request: error reading a body from connection: bytes remaining on stream
2025-01-09T00:03:29-05:00: POST /fixed_chunk: 400 Bad Request: error reading a body from connection: bytes remaining on stream
2025-01-09T00:03:29-05:00: POST /fixed_chunk: 400 Bad Request: error reading a body from connection: bytes remaining on stream
2025-01-09T00:03:29-05:00: POST /fixed_chunk: 400 Bad Request: error reading a body from connection: bytes remaining on stream
2025-01-09T00:03:29-05:00: POST /fixed_chunk: 400 Bad Request: error reading a body from connection: bytes remaining on stream
2025-01-09T00:03:29-05:00: POST /fixed_chunk: 400 Bad Request: error reading a body from connection: bytes remaining on stream
2025-01-09T00:03:29-05:00: POST /fixed_chunk: 400 Bad Request: error reading a body from connection: bytes remaining on stream
2025-01-09T00:03:29-05:00: backup failed: connection error: bytes remaining on stream
2025-01-09T00:03:29-05:00: removing failed backup
2025-01-09T00:03:29-05:00: removing backup snapshot "/mnt/datastore/backupPool/vm/401/2025-01-09T04:58:11Z"
2025-01-09T00:03:29-05:00: POST /fixed_chunk: 400 Bad Request: error reading a body from connection: bytes remaining on stream
2025-01-09T00:03:29-05:00: TASK ERROR: connection error: bytes remaining on stream

iperf3 test below. speeds aren't the greatest but i'd still expect the backup to be able to run.
PBS is 10.95.3.3 in this case
PVE B is 10.4.19.3

root@PVE-B:~# iperf3 -c 10.95.3.3
Connecting to host 10.95.3.3, port 5201
[  5] local 10.4.19.3 port 59420 connected to 10.95.3.3 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  1023 KBytes  8.38 Mbits/sec    6   40.1 KBytes       
[  5]   1.00-2.00   sec   737 KBytes  6.04 Mbits/sec    1   40.1 KBytes       
[  5]   2.00-3.00   sec   615 KBytes  5.03 Mbits/sec    1   37.4 KBytes       
[  5]   3.00-4.00   sec   615 KBytes  5.03 Mbits/sec    2   36.1 KBytes       
[  5]   4.00-5.00   sec   615 KBytes  5.03 Mbits/sec    0   48.1 KBytes       
[  5]   5.00-6.00   sec   860 KBytes  7.05 Mbits/sec    2   28.1 KBytes       
[  5]   6.00-7.00   sec   615 KBytes  5.03 Mbits/sec    0   41.4 KBytes       
[  5]   7.00-8.00   sec   737 KBytes  6.04 Mbits/sec    1   36.1 KBytes       
[  5]   8.00-9.00   sec   615 KBytes  5.03 Mbits/sec    1   33.4 KBytes       
^C[  5]   9.00-9.95   sec   492 KBytes  4.23 Mbits/sec    1   29.4 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-9.95   sec  6.76 MBytes  5.70 Mbits/sec   15             sender
[  5]   0.00-9.95   sec  0.00 Bytes  0.00 bits/sec                  receiver
iperf3: interrupt - the client has terminated
root@PVE-B:~#

output of proxmox-backup-client benchmark --repository backupUser@pbs@10.95.3.3:backupPool from PVE B shell below:

Uploaded 14 chunks in 40 seconds.
Time per request: 2926272 microseconds.
TLS speed: 1.43 MB/s   
SHA256 speed: 2148.01 MB/s   
Compression speed: 557.38 MB/s   
Decompress speed: 719.52 MB/s   
AES256/GCM speed: 4602.96 MB/s   
Verify speed: 539.10 MB/s   
┌───────────────────────────────────┬─────────────────────┐
│ Name                              │ Value               │
╞═══════════════════════════════════╪═════════════════════╡
│ TLS (maximal backup upload speed) │ 1.43 MB/s (0%)      │
├───────────────────────────────────┼─────────────────────┤
│ SHA256 checksum computation speed │ 2148.01 MB/s (106%) │
├───────────────────────────────────┼─────────────────────┤
│ ZStd level 1 compression speed    │ 557.38 MB/s (74%)   │
├───────────────────────────────────┼─────────────────────┤
│ ZStd level 1 decompression speed  │ 719.52 MB/s (60%)   │
├───────────────────────────────────┼─────────────────────┤
│ Chunk verification speed          │ 539.10 MB/s (71%)   │
├───────────────────────────────────┼─────────────────────┤
│ AES256 GCM encryption speed       │ 4602.96 MB/s (126%) │
└───────────────────────────────────┴─────────────────────┘

 

[fabian]

does a host or container backup work?

 

[marc_things]

@fabian i was able to backup an ubuntu 24.02 container over VPN: 64GB disk, 512 RAM.

i'm unaware to do a host backup to pbs. i tried looking for documentation but couldn't find any. i'm happy to try if you can share the docs.

I tried VM again and still running into the same issues.

 

[marc_things]

i was a bit more patient this time and it does look like the VM is starting to backup over VPN. just extremely slow. after 12 minutes:

NFO:   0% (228.0 MiB of 120.0 GiB) in 3s, read: 76.0 MiB/s, write: 36.0 MiB/s
INFO:   1% (1.2 GiB of 120.0 GiB) in 9m 30s, read: 1.8 MiB/s, write: 1.7 MiB/s
INFO:   2% (2.7 GiB of 120.0 GiB) in 11m 44s, read: 11.7 MiB/s, write: 2.9 MiB/s

i'm assuming the answer is i can't speed it up, but is there anything i can tweak to try and make this a bit faster?

 

[LnxBil]

i was a bit more patient this time and it does look like the VM is starting to backup over VPN. just extremely slow. after 12 minutes:

NFO:   0% (228.0 MiB of 120.0 GiB) in 3s, read: 76.0 MiB/s, write: 36.0 MiB/s
INFO:   1% (1.2 GiB of 120.0 GiB) in 9m 30s, read: 1.8 MiB/s, write: 1.7 MiB/s
INFO:   2% (2.7 GiB of 120.0 GiB) in 11m 44s, read: 11.7 MiB/s, write: 2.9 MiB/s

i'm assuming the answer is i can't speed it up, but is there anything i can tweak to try and make this a bit faster?

The speed is acutually faster than benchmarked and is consistent with your iperf benchmark. So what are you expecting more here? The line / connection is the limit.

i'm unaware to do a host backup to pbs. i tried looking for documentation but couldn't find any. i'm happy to try if you can share the docs.

The "normal" proxmox backup client is capable of doing a host backup.

 

[marc_things]

@LnxBil thanks for the additional info. I was thinking maybe there was a way to reduce some overhead to allow larger chunks. I'm satisfied that its working though! I'll check out the host backup info and mark this post as SOLVED.