I have 2 PVE hosts and one PBS host.
PVE A is located at the same location as PBS and works fine.
PVE B is at a second location and backups seem to get stuck at 0% and never progress.
Both sites are running pfsense firewalls.
I have a wireguard site-to-site tunnel between each.
MTU & MSS on both pfSense wireguard interfaces is set to 1420. I've tried 1200 as well with the same results.
I was able to add PBS datastore to PVE B without any issues but when I try to backup it gets to 0% and just hangs there.
My goal is to be able to backup VMs hosted on PVE B to the PBS server over the wireguard site to site tunnel.
here is the info from the output on the Backup task summary on PBS:
iperf3 test below. speeds aren't the greatest but i'd still expect the backup to be able to run.
PBS is 10.95.3.3 in this case
PVE B is 10.4.19.3
output of
PVE A is located at the same location as PBS and works fine.
PVE B is at a second location and backups seem to get stuck at 0% and never progress.
Both sites are running pfsense firewalls.
I have a wireguard site-to-site tunnel between each.
MTU & MSS on both pfSense wireguard interfaces is set to 1420. I've tried 1200 as well with the same results.
I was able to add PBS datastore to PVE B without any issues but when I try to backup it gets to 0% and just hangs there.
My goal is to be able to backup VMs hosted on PVE B to the PBS server over the wireguard site to site tunnel.
Code:
INFO: starting new backup job: vzdump 401 --mode snapshot --notes-template '{{guestname}}' --storage backupPool --remove 0 --node beemox --notification-mode auto
INFO: Starting Backup of VM 401 (qemu)
INFO: Backup started at 2025-01-08 23:58:11
INFO: status = running
INFO: VM Name: RA002S
INFO: include disk 'scsi0' 'beepool:vm-401-disk-0' 120G
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating Proxmox Backup Server archive 'vm/401/2025-01-09T04:58:11Z'
INFO: enabling encryption
INFO: issuing guest-agent 'fs-freeze' command
INFO: issuing guest-agent 'fs-thaw' command
INFO: started backup task '0d899831-994f-40e2-b56a-3f0be22c2122'
INFO: resuming VM again
INFO: scsi0: dirty-bitmap status: existing bitmap was invalid and has been cleared
INFO: 0% (212.0 MiB of 120.0 GiB) in 3s, read: 70.7 MiB/s, write: 30.7 MiB/shere is the info from the output on the Backup task summary on PBS:
Code:
2025-01-08T23:58:12-05:00: GET /previous: 400 Bad Request: no valid previous backup
2025-01-08T23:58:12-05:00: created new fixed index 1 ("vm/401/2025-01-09T04:58:11Z/drive-scsi0.img.fidx")
2025-01-08T23:58:12-05:00: add blob "/mnt/datastore/backupPool/vm/401/2025-01-09T04:58:11Z/qemu-server.conf.blob" (404 bytes, comp: 404)
2025-01-09T00:03:29-05:00: POST /fixed_chunk: 400 Bad Request: error reading a body from connection: bytes remaining on stream
2025-01-09T00:03:29-05:00: POST /fixed_chunk: 400 Bad Request: error reading a body from connection: bytes remaining on stream
2025-01-09T00:03:29-05:00: POST /fixed_chunk: 400 Bad Request: error reading a body from connection: bytes remaining on stream
2025-01-09T00:03:29-05:00: POST /fixed_chunk: 400 Bad Request: error reading a body from connection: bytes remaining on stream
2025-01-09T00:03:29-05:00: POST /fixed_chunk: 400 Bad Request: error reading a body from connection: bytes remaining on stream
2025-01-09T00:03:29-05:00: POST /fixed_chunk: 400 Bad Request: error reading a body from connection: bytes remaining on stream
2025-01-09T00:03:29-05:00: POST /fixed_chunk: 400 Bad Request: error reading a body from connection: bytes remaining on stream
2025-01-09T00:03:29-05:00: POST /fixed_chunk: 400 Bad Request: error reading a body from connection: bytes remaining on stream
2025-01-09T00:03:29-05:00: backup failed: connection error: bytes remaining on stream
2025-01-09T00:03:29-05:00: removing failed backup
2025-01-09T00:03:29-05:00: removing backup snapshot "/mnt/datastore/backupPool/vm/401/2025-01-09T04:58:11Z"
2025-01-09T00:03:29-05:00: POST /fixed_chunk: 400 Bad Request: error reading a body from connection: bytes remaining on stream
2025-01-09T00:03:29-05:00: TASK ERROR: connection error: bytes remaining on streamiperf3 test below. speeds aren't the greatest but i'd still expect the backup to be able to run.
PBS is 10.95.3.3 in this case
PVE B is 10.4.19.3
Code:
root@PVE-B:~# iperf3 -c 10.95.3.3
Connecting to host 10.95.3.3, port 5201
[ 5] local 10.4.19.3 port 59420 connected to 10.95.3.3 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 1023 KBytes 8.38 Mbits/sec 6 40.1 KBytes
[ 5] 1.00-2.00 sec 737 KBytes 6.04 Mbits/sec 1 40.1 KBytes
[ 5] 2.00-3.00 sec 615 KBytes 5.03 Mbits/sec 1 37.4 KBytes
[ 5] 3.00-4.00 sec 615 KBytes 5.03 Mbits/sec 2 36.1 KBytes
[ 5] 4.00-5.00 sec 615 KBytes 5.03 Mbits/sec 0 48.1 KBytes
[ 5] 5.00-6.00 sec 860 KBytes 7.05 Mbits/sec 2 28.1 KBytes
[ 5] 6.00-7.00 sec 615 KBytes 5.03 Mbits/sec 0 41.4 KBytes
[ 5] 7.00-8.00 sec 737 KBytes 6.04 Mbits/sec 1 36.1 KBytes
[ 5] 8.00-9.00 sec 615 KBytes 5.03 Mbits/sec 1 33.4 KBytes
^C[ 5] 9.00-9.95 sec 492 KBytes 4.23 Mbits/sec 1 29.4 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-9.95 sec 6.76 MBytes 5.70 Mbits/sec 15 sender
[ 5] 0.00-9.95 sec 0.00 Bytes 0.00 bits/sec receiver
iperf3: interrupt - the client has terminated
root@PVE-B:~#output of
proxmox-backup-client benchmark --repository backupUser@pbs@10.95.3.3:backupPool from PVE B shell below:
Code:
Uploaded 14 chunks in 40 seconds.
Time per request: 2926272 microseconds.
TLS speed: 1.43 MB/s
SHA256 speed: 2148.01 MB/s
Compression speed: 557.38 MB/s
Decompress speed: 719.52 MB/s
AES256/GCM speed: 4602.96 MB/s
Verify speed: 539.10 MB/s
┌───────────────────────────────────┬─────────────────────┐
│ Name │ Value │
╞═══════════════════════════════════╪═════════════════════╡
│ TLS (maximal backup upload speed) │ 1.43 MB/s (0%) │
├───────────────────────────────────┼─────────────────────┤
│ SHA256 checksum computation speed │ 2148.01 MB/s (106%) │
├───────────────────────────────────┼─────────────────────┤
│ ZStd level 1 compression speed │ 557.38 MB/s (74%) │
├───────────────────────────────────┼─────────────────────┤
│ ZStd level 1 decompression speed │ 719.52 MB/s (60%) │
├───────────────────────────────────┼─────────────────────┤
│ Chunk verification speed │ 539.10 MB/s (71%) │
├───────────────────────────────────┼─────────────────────┤
│ AES256 GCM encryption speed │ 4602.96 MB/s (126%) │
└───────────────────────────────────┴─────────────────────┘