I have a machine with a few VMs running Proxmox 7.1-5. I have backup jobs running to a dedicated PBS running Backup Server 2.0-14. Around 50% of the time, the backups fail due to write errors. I'm not sure if the PBS is running into bandwidth issues writing or if there is another issue.
The proxmox server has dual xeon w/256Gb ram and raid controller in IT mode. VM disks are on raidZ1 and it has 10Gb connection. PBS is an AMD FX6100 w/8Gb ram and raid controller in IT mode w/1Gb connnection. I've had no problems restoring VMs from the PBS. During backups, the CPU usage may go to 50%, but RAM doesn't go much over 5Gb usage.
This happens at different times of the backup process.
Examples:
INFO: 5% (1.8 GiB of 32.0 GiB) in 15s, read: 112.0 MiB/s, write: 14.7 MiB/s
INFO: 5% (1.9 GiB of 32.0 GiB) in 1m 33s, read: 1.4 MiB/s, write: 105.0 KiB/s
ERROR: backup write data failed: command error: write_data upload error: pipelined request failed: broken pipe
INFO: aborting backup job
-------------------------------------------------------------------------------------
INFO: 47% (15.2 GiB of 32.0 GiB) in 2m 33s, read: 90.4 MiB/s, write: 27.2 MiB/s
INFO: 48% (15.4 GiB of 32.0 GiB) in 2m 36s, read: 80.0 MiB/s, write: 65.3 MiB/s
INFO: 48% (15.6 GiB of 32.0 GiB) in 3m 58s, read: 1.9 MiB/s, write: 1.2 MiB/s
ERROR: backup write data failed: command error: protocol canceled
INFO: aborting backup job
Could this be the proxmox backup client tossing out data too fast for the PBS to consume?
root@pbs:~# proxmox-backup-client benchmark
┌───────────────────────────────────┬────────────────────┐
│ Name │ Value │
╞═══════════════════════════════════╪════════════════════╡
│ TLS (maximal backup upload speed) │ 472.29 MB/s (38%) │
├───────────────────────────────────┼────────────────────┤
│ SHA256 checksum computation speed │ 286.59 MB/s (14%) │
├───────────────────────────────────┼────────────────────┤
│ ZStd level 1 compression speed │ 402.73 MB/s (54%) │
├───────────────────────────────────┼────────────────────┤
│ ZStd level 1 decompression speed │ 571.89 MB/s (48%) │
├───────────────────────────────────┼────────────────────┤
│ Chunk verification speed │ 194.12 MB/s (26%) │
├───────────────────────────────────┼────────────────────┤
│ AES256 GCM encryption speed │ 1027.40 MB/s (28%) │
└───────────────────────────────────┴────────────────────┘
The proxmox server has dual xeon w/256Gb ram and raid controller in IT mode. VM disks are on raidZ1 and it has 10Gb connection. PBS is an AMD FX6100 w/8Gb ram and raid controller in IT mode w/1Gb connnection. I've had no problems restoring VMs from the PBS. During backups, the CPU usage may go to 50%, but RAM doesn't go much over 5Gb usage.
This happens at different times of the backup process.
Examples:
INFO: 5% (1.8 GiB of 32.0 GiB) in 15s, read: 112.0 MiB/s, write: 14.7 MiB/s
INFO: 5% (1.9 GiB of 32.0 GiB) in 1m 33s, read: 1.4 MiB/s, write: 105.0 KiB/s
ERROR: backup write data failed: command error: write_data upload error: pipelined request failed: broken pipe
INFO: aborting backup job
-------------------------------------------------------------------------------------
INFO: 47% (15.2 GiB of 32.0 GiB) in 2m 33s, read: 90.4 MiB/s, write: 27.2 MiB/s
INFO: 48% (15.4 GiB of 32.0 GiB) in 2m 36s, read: 80.0 MiB/s, write: 65.3 MiB/s
INFO: 48% (15.6 GiB of 32.0 GiB) in 3m 58s, read: 1.9 MiB/s, write: 1.2 MiB/s
ERROR: backup write data failed: command error: protocol canceled
INFO: aborting backup job
Could this be the proxmox backup client tossing out data too fast for the PBS to consume?
root@pbs:~# proxmox-backup-client benchmark
┌───────────────────────────────────┬────────────────────┐
│ Name │ Value │
╞═══════════════════════════════════╪════════════════════╡
│ TLS (maximal backup upload speed) │ 472.29 MB/s (38%) │
├───────────────────────────────────┼────────────────────┤
│ SHA256 checksum computation speed │ 286.59 MB/s (14%) │
├───────────────────────────────────┼────────────────────┤
│ ZStd level 1 compression speed │ 402.73 MB/s (54%) │
├───────────────────────────────────┼────────────────────┤
│ ZStd level 1 decompression speed │ 571.89 MB/s (48%) │
├───────────────────────────────────┼────────────────────┤
│ Chunk verification speed │ 194.12 MB/s (26%) │
├───────────────────────────────────┼────────────────────┤
│ AES256 GCM encryption speed │ 1027.40 MB/s (28%) │
└───────────────────────────────────┴────────────────────┘
Last edited:
