Error when backing up to Wasabi S3

[cglmicro]

Hello all,
When I try to do a backup to a Wasabi S3 storage from within PVE, on two different nodes in my cluster, and two different VMs, I get this same error:

INFO: starting new backup job: vzdump 23071 --storage proxmox-cgl-s3-1 --notification-mode auto --node proxmox26s --remove 0 --mode stop --notes-template '{{guestname}}' --compress zstd
INFO: Starting Backup of VM 23071 (qemu)
INFO: Backup started at 2025-05-13 20:34:15
INFO: status = stopped
INFO: backup mode: stop
INFO: ionice priority: 7
INFO: VM Name: win11-test3
INFO: include disk 'ide0' 'local:23071/vm-23071-disk-1.qcow2' 80G
INFO: include disk 'efidisk0' 'local:23071/vm-23071-disk-0.qcow2' 528K
INFO: include disk 'tpmstate0' 'local:23071/vm-23071-disk-2.raw' 4M
INFO: creating vzdump archive '/mnt/proxmox-cgl-s3-1/dump/vzdump-qemu-23071-2025_05_13-20_34_15.vma.zst'
INFO: starting kvm to execute backup task
swtpm_setup: Not overwriting existing state file.
INFO: attaching TPM drive to QEMU for backup
INFO: started backup task 'e8ccf27c-e0e7-4f2a-bc9f-9b7e38e58e9d'
INFO:   1% (1.1 GiB of 80.0 GiB) in 3s, read: 373.4 MiB/s, write: 330.9 MiB/s
INFO:   2% (2.2 GiB of 80.0 GiB) in 6s, read: 378.2 MiB/s, write: 323.3 MiB/s
INFO:   4% (3.7 GiB of 80.0 GiB) in 9s, read: 502.7 MiB/s, write: 404.8 MiB/s
INFO:   6% (4.9 GiB of 80.0 GiB) in 13s, read: 318.1 MiB/s, write: 270.5 MiB/s
INFO:   7% (6.1 GiB of 80.0 GiB) in 16s, read: 390.3 MiB/s, write: 356.6 MiB/s
INFO:   9% (7.3 GiB of 80.0 GiB) in 19s, read: 418.1 MiB/s, write: 362.4 MiB/s
INFO:  10% (8.6 GiB of 80.0 GiB) in 22s, read: 447.1 MiB/s, write: 379.6 MiB/s
INFO:  12% (9.8 GiB of 80.0 GiB) in 25s, read: 420.5 MiB/s, write: 357.4 MiB/s
INFO:  14% (11.3 GiB of 80.0 GiB) in 28s, read: 494.2 MiB/s, write: 301.8 MiB/s
INFO:  15% (12.3 GiB of 80.0 GiB) in 1m 18s, read: 21.6 MiB/s, write: 11.5 MiB/s
INFO:  17% (14.1 GiB of 80.0 GiB) in 1m 21s, read: 607.8 MiB/s, write: 338.5 MiB/s
INFO:  19% (15.5 GiB of 80.0 GiB) in 1m 24s, read: 473.7 MiB/s, write: 373.0 MiB/s
INFO:  21% (16.8 GiB of 80.0 GiB) in 1m 27s, read: 456.8 MiB/s, write: 369.8 MiB/s
INFO:  22% (18.0 GiB of 80.0 GiB) in 1m 30s, read: 413.2 MiB/s, write: 347.5 MiB/s
INFO:  24% (19.5 GiB of 80.0 GiB) in 1m 33s, read: 511.1 MiB/s, write: 356.5 MiB/s
INFO:  26% (21.4 GiB of 80.0 GiB) in 1m 36s, read: 619.9 MiB/s, write: 366.0 MiB/s
INFO:  30% (24.1 GiB of 80.0 GiB) in 1m 39s, read: 947.3 MiB/s, write: 423.2 MiB/s
INFO:  33% (27.0 GiB of 80.0 GiB) in 1m 42s, read: 963.0 MiB/s, write: 354.3 MiB/s
INFO:  37% (29.9 GiB of 80.0 GiB) in 1m 45s, read: 1016.2 MiB/s, write: 275.2 MiB/s
zstd: error 70 : Write error : cannot write block : Input/output error
INFO:  37% (29.9 GiB of 80.0 GiB) in 4m 10s, read: 0 B/s, write: 0 B/s
Warning: unable to close filehandle GEN30793 properly: Input/output error at /usr/share/perl5/PVE/VZDump/QemuServer.pm line 1011.
ERROR: vma_queue_write: write error - Broken pipe
INFO: aborting backup job
INFO: stopping kvm after backup task
ERROR: Backup of VM 23071 failed - vma_queue_write: write error - Broken pipe
INFO: Failed at 2025-05-13 20:39:39
INFO: Backup job finished with errors
INFO: notified via target `mail-to-root`
TASK ERROR: job errors

and:

INFO: starting new backup job: vzdump 14061 --compress zstd --mode snapshot --storage proxmox-cgl-s3-1 --notification-mode auto --node proxmox18s --notes-template '{{guestname}}' --remove 0
INFO: Starting Backup of VM 14061 (qemu)
INFO: Backup started at 2025-05-13 19:56:34
INFO: status = running
INFO: VM Name: screenconnect.legardeur.net
INFO: include disk 'ide0' 'local:14061/vm-14061-disk-0.qcow2' 120G
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating vzdump archive '/mnt/proxmox-cgl-s3-1/dump/vzdump-qemu-14061-2025_05_13-19_56_34.vma.zst'
INFO: issuing guest-agent 'fs-freeze' command
INFO: issuing guest-agent 'fs-thaw' command
INFO: started backup task 'ac4cecc7-17ff-4db6-9753-88a959945f72'
INFO: resuming VM again
INFO:   0% (1.1 GiB of 120.0 GiB) in 4s, read: 271.0 MiB/s, write: 217.5 MiB/s
INFO:   1% (1.8 GiB of 120.0 GiB) in 7s, read: 236.8 MiB/s, write: 212.1 MiB/s
INFO:   2% (2.4 GiB of 120.0 GiB) in 10s, read: 232.0 MiB/s, write: 201.4 MiB/s
INFO:   3% (3.9 GiB of 120.0 GiB) in 16s, read: 246.5 MiB/s, write: 228.3 MiB/s
INFO:   4% (5.0 GiB of 120.0 GiB) in 20s, read: 290.9 MiB/s, write: 266.0 MiB/s
INFO:   5% (6.1 GiB of 120.0 GiB) in 24s, read: 279.1 MiB/s, write: 262.5 MiB/s
INFO:   6% (7.3 GiB of 120.0 GiB) in 30s, read: 212.7 MiB/s, write: 178.3 MiB/s
INFO:   7% (8.6 GiB of 120.0 GiB) in 35s, read: 256.6 MiB/s, write: 219.2 MiB/s
INFO:   8% (9.7 GiB of 120.0 GiB) in 1m 25s, read: 22.2 MiB/s, write: 19.7 MiB/s
INFO:   9% (11.1 GiB of 120.0 GiB) in 1m 30s, read: 298.0 MiB/s, write: 269.2 MiB/s
INFO:  10% (12.2 GiB of 120.0 GiB) in 1m 35s, read: 218.2 MiB/s, write: 192.0 MiB/s
INFO:  11% (13.4 GiB of 120.0 GiB) in 1m 39s, read: 313.6 MiB/s, write: 246.2 MiB/s
INFO:  12% (14.4 GiB of 120.0 GiB) in 1m 43s, read: 258.3 MiB/s, write: 198.0 MiB/s
INFO:  13% (15.7 GiB of 120.0 GiB) in 1m 47s, read: 309.9 MiB/s, write: 241.6 MiB/s
INFO:  14% (17.0 GiB of 120.0 GiB) in 1m 52s, read: 274.8 MiB/s, write: 208.3 MiB/s
INFO:  15% (18.1 GiB of 120.0 GiB) in 1m 57s, read: 227.6 MiB/s, write: 180.4 MiB/s
zstd: error 70 : Write error : cannot write block : Input/output error
INFO:  15% (18.2 GiB of 120.0 GiB) in 4m 10s, read: 865.2 KiB/s, write: 695.3 KiB/s
Warning: unable to close filehandle GEN10011 properly: Input/output error at /usr/share/perl5/PVE/VZDump/QemuServer.pm line 1011.
ERROR: vma_queue_write: write error - Broken pipe
INFO: aborting backup job
INFO: resuming VM again
ERROR: Backup of VM 14061 failed - vma_queue_write: write error - Broken pipe
INFO: Failed at 2025-05-13 20:01:59
INFO: Backup job finished with errors
INFO: notified via target `mail-to-root`
TASK ERROR: job errors

I opened a ticket with Wasabi, and they said there are no errors on their side, and no throttle rule to limit after a few Gb/seconds.

I also tried without compression,

I finally tried to limit bandwidth in /etc/vzdump.conf:

#bwlimit in KiB/s:
bwlimit: 50000

But it freeze at the same place for VM 14061, I'm testing VM 23071 tonight and will be back with results, but it slowing down a LOT after 70%.

How to know witch side is the cause?

Thank you.

 

[fabian]

maybe there's a limit how big a single file can get? note that backing up to WAN storage without fleecing is likely to break I/O and thus execution of your guest

 

[cglmicro]

maybe there's a limit how big a single file can get? note that backing up to WAN storage without fleecing is likely to break I/O and thus execution of your guest

It make sense, I'll ask Wasabi support.
Thank you for the suggestion.

 

[cglmicro]

...

But it freeze at the same place for VM 14061, I'm testing VM 23071 tonight and will be back with results, but it slowing down a LOT after 70%.

...

I did the test, and it returned an error when it reached 100% also for this VM with a very low bandwidth.
I'll come back if I find a solution.
Thank you.