Slow backup with PBS

[onlyzeros]

Hi all,
I'm having issues backing up PVE to PBS, it's very slow and inconsistent write speed.
To backup a VM with 300Gib disk takes over an hour.


PVE - Lenovo P520
Xeon(R) W-2255 CPU
190 GiB DIMM DDR4 2666 MHz
GeForce RTX 4060 Ti 16GB
Vms data are stored in ZFS 3x SSD SanDisk X300 DC Raidz1

PBS - Terramaster F4-424
Running as VM in Truenas scale
CPU - 4 cores allocated
Ram - 8GiB DDR5 allocated
Boot disk - nvme zvol Volume Size:50 GiB
Datastore - zvol Volume Size: 3 TiB on a 4x 10TB in Raidz2

root@homelab:~# proxmox-backup-client benchmark
SHA256 speed: 555.12 MB/s  
Compression speed: 594.51 MB/s  
Decompress speed: 930.69 MB/s  
AES256/GCM speed: 4009.89 MB/s  
Verify speed: 343.80 MB/s  
┌───────────────────────────────────┬─────────────────────┐
│ Name                              │ Value               │
╞═══════════════════════════════════╪═════════════════════╡
│ TLS (maximal backup upload speed) │ not tested          │
├───────────────────────────────────┼─────────────────────┤
│ SHA256 checksum computation speed │ 555.12 MB/s (27%)   │
├───────────────────────────────────┼─────────────────────┤
│ ZStd level 1 compression speed    │ 594.51 MB/s (79%)   │
├───────────────────────────────────┼─────────────────────┤
│ ZStd level 1 decompression speed  │ 930.69 MB/s (78%)   │
├───────────────────────────────────┼─────────────────────┤
│ Chunk verification speed          │ 343.80 MB/s (45%)   │
├───────────────────────────────────┼─────────────────────┤
│ AES256 GCM encryption speed       │ 4009.89 MB/s (110%) │
└───────────────────────────────────┴─────────────────────┘
root@homelab:~# iperf3 -c 192.168.1.6
Connecting to host 192.168.1.6, port 5201
[  5] local 192.168.1.39 port 51246 connected to 192.168.1.6 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   239 MBytes  2.00 Gbits/sec    0    539 KBytes      
[  5]   1.00-2.00   sec   234 MBytes  1.96 Gbits/sec    0    655 KBytes      
[  5]   2.00-3.00   sec   238 MBytes  1.99 Gbits/sec    0    803 KBytes      
[  5]   3.00-4.00   sec   232 MBytes  1.95 Gbits/sec    0    946 KBytes      
[  5]   4.00-5.00   sec   234 MBytes  1.96 Gbits/sec    0    993 KBytes      
[  5]   5.00-6.00   sec   238 MBytes  1.99 Gbits/sec    0    993 KBytes      
[  5]   6.00-7.00   sec   235 MBytes  1.97 Gbits/sec    0    993 KBytes      
[  5]   7.00-8.00   sec   234 MBytes  1.96 Gbits/sec    0    993 KBytes      
[  5]   8.00-9.00   sec   239 MBytes  2.00 Gbits/sec    0    993 KBytes      
[  5]   9.00-10.00  sec   239 MBytes  2.00 Gbits/sec    0    993 KBytes      
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  2.31 GBytes  1.98 Gbits/sec    0             sender
[  5]   0.00-10.00  sec  2.30 GBytes  1.98 Gbits/sec                  receiver

INFO: starting new backup job: vzdump 108 --notes-template '{{guestname}}' --node homelab --mode stop --notification-mode auto --storage puddle --remove 0
INFO: Starting Backup of VM 108 (qemu)
INFO: Backup started at 2025-01-26 08:00:50
INFO: status = stopped
INFO: backup mode: stop
INFO: ionice priority: 7
INFO: VM Name: windows11
INFO: exclude disk 'scsi0' 'test:0.0.1.scsi-36589cfc000000e743858e34c9cf6edc4' (backup=no)
INFO: include disk 'sata0' 'vms-pool:vm-108-disk-3' 306G
INFO: exclude disk 'sata1' 'nas1-games:0.0.1.scsi-36589cfc000000e743858e34c9cf6edc4' (backup=no)
INFO: include disk 'efidisk0' 'vms-pool:vm-108-disk-2' 1M
INFO: include disk 'tpmstate0' 'vms-pool:vm-108-disk-4' 4M
INFO: creating Proxmox Backup Server archive 'vm/108/2025-01-26T08:00:50Z'
INFO: starting kvm to execute backup task
swtpm_setup: Not overwriting existing state file.
INFO: attaching TPM drive to QEMU for backup
INFO: started backup task '6571d477-8c4d-40df-8de5-cb544b6e9200'
INFO: efidisk0: dirty-bitmap status: created new
INFO: sata0: dirty-bitmap status: created new
INFO: tpmstate0-backup: dirty-bitmap status: created new
INFO:   0% (636.0 MiB of 306.0 GiB) in 3s, read: 212.0 MiB/s, write: 169.3 MiB/s
INFO:   1% (3.1 GiB of 306.0 GiB) in 53s, read: 51.4 MiB/s, write: 40.6 MiB/s
INFO:   2% (6.2 GiB of 306.0 GiB) in 1m 26s, read: 94.3 MiB/s, write: 80.1 MiB/s
INFO:   3% (9.3 GiB of 306.0 GiB) in 1m 46s, read: 162.4 MiB/s, write: 143.6 MiB/s
INFO:   4% (12.4 GiB of 306.0 GiB) in 1m 58s, read: 258.7 MiB/s, write: 157.0 MiB/s
INFO:   5% (15.5 GiB of 306.0 GiB) in 2m 28s, read: 108.4 MiB/s, write: 26.5 MiB/s
INFO:   6% (18.4 GiB of 306.0 GiB) in 3m 16s, read: 60.4 MiB/s, write: 36.6 MiB/s
INFO:   7% (21.4 GiB of 306.0 GiB) in 4m 8s, read: 60.2 MiB/s, write: 30.0 MiB/s
INFO:   8% (24.6 GiB of 306.0 GiB) in 4m 31s, read: 138.6 MiB/s, write: 18.6 MiB/s
INFO:   9% (27.7 GiB of 306.0 GiB) in 4m 40s, read: 354.2 MiB/s, write: 24.4 MiB/s
INFO:  10% (30.6 GiB of 306.0 GiB) in 5m 3s, read: 131.3 MiB/s, write: 43.0 MiB/s
INFO:  11% (33.7 GiB of 306.0 GiB) in 12m 1s, read: 7.5 MiB/s, write: 4.6 MiB/s
INFO:  12% (36.7 GiB of 306.0 GiB) in 19m 51s, read: 6.7 MiB/s, write: 5.9 MiB/s
INFO:  13% (39.8 GiB of 306.0 GiB) in 26m 34s, read: 7.8 MiB/s, write: 6.5 MiB/s
INFO:  14% (43.4 GiB of 306.0 GiB) in 29m 20s, read: 22.4 MiB/s, write: 6.0 MiB/s
INFO:  15% (46.2 GiB of 306.0 GiB) in 29m 52s, read: 87.5 MiB/s, write: 5.4 MiB/s
INFO:  16% (49.6 GiB of 306.0 GiB) in 30m, read: 433.5 MiB/s, write: 20.5 MiB/s
INFO:  17% (52.2 GiB of 306.0 GiB) in 31m 21s, read: 33.1 MiB/s, write: 9.2 MiB/s
INFO:  18% (55.5 GiB of 306.0 GiB) in 32m 26s, read: 51.7 MiB/s, write: 189.0 KiB/s
INFO:  19% (58.3 GiB of 306.0 GiB) in 32m 32s, read: 476.7 MiB/s, write: 0 B/s
INFO:  20% (61.5 GiB of 306.0 GiB) in 32m 39s, read: 480.0 MiB/s, write: 0 B/s
INFO:  21% (64.3 GiB of 306.0 GiB) in 32m 45s, read: 477.3 MiB/s, write: 0 B/s
INFO:  22% (67.6 GiB of 306.0 GiB) in 32m 52s, read: 474.9 MiB/s, write: 0 B/s
INFO:  23% (70.8 GiB of 306.0 GiB) in 32m 59s, read: 474.3 MiB/s, write: 0 B/s
INFO:  24% (73.6 GiB of 306.0 GiB) in 33m 5s, read: 474.7 MiB/s, write: 0 B/s
INFO:  25% (76.9 GiB of 306.0 GiB) in 33m 12s, read: 476.0 MiB/s, write: 0 B/s
INFO:  26% (79.6 GiB of 306.0 GiB) in 33m 18s, read: 476.0 MiB/s, write: 18.7 MiB/s
INFO:  27% (83.3 GiB of 306.0 GiB) in 37m 32s, read: 14.6 MiB/s, write: 7.3 MiB/s
INFO:  28% (86.2 GiB of 306.0 GiB) in 38m 17s, read: 67.3 MiB/s, write: 12.1 MiB/s
INFO:  30% (92.1 GiB of 306.0 GiB) in 38m 20s, read: 1.9 GiB/s, write: 37.3 MiB/s
INFO:  31% (95.1 GiB of 306.0 GiB) in 38m 28s, read: 389.0 MiB/s, write: 8.0 MiB/s
INFO:  32% (98.1 GiB of 306.0 GiB) in 38m 32s, read: 763.0 MiB/s, write: 19.0 MiB/s
INFO:  33% (101.1 GiB of 306.0 GiB) in 38m 45s, read: 238.5 MiB/s, write: 9.2 MiB/s
INFO:  34% (104.3 GiB of 306.0 GiB) in 38m 49s, read: 814.0 MiB/s, write: 13.0 MiB/s
INFO:  35% (107.6 GiB of 306.0 GiB) in 38m 55s, read: 564.7 MiB/s, write: 8.7 MiB/s
INFO:  36% (110.6 GiB of 306.0 GiB) in 39m, read: 620.0 MiB/s, write: 12.8 MiB/s
INFO:  37% (113.5 GiB of 306.0 GiB) in 39m 6s, read: 489.3 MiB/s, write: 3.3 MiB/s
INFO:  38% (116.6 GiB of 306.0 GiB) in 39m 13s, read: 449.7 MiB/s, write: 22.9 MiB/s
INFO:  39% (119.5 GiB of 306.0 GiB) in 39m 19s, read: 504.0 MiB/s, write: 682.7 KiB/s
INFO:  40% (122.6 GiB of 306.0 GiB) in 39m 37s, read: 175.8 MiB/s, write: 19.1 MiB/s
INFO:  41% (125.5 GiB of 306.0 GiB) in 39m 44s, read: 418.9 MiB/s, write: 1.1 MiB/s
INFO:  42% (128.6 GiB of 306.0 GiB) in 39m 50s, read: 532.7 MiB/s, write: 1.3 MiB/s
INFO:  43% (131.8 GiB of 306.0 GiB) in 39m 57s, read: 476.0 MiB/s, write: 1.1 MiB/s
INFO:  44% (135.1 GiB of 306.0 GiB) in 40m 4s, read: 475.4 MiB/s, write: 1.1 MiB/s
INFO:  45% (137.7 GiB of 306.0 GiB) in 40m 9s, read: 541.6 MiB/s, write: 819.2 KiB/s
INFO:  46% (141.1 GiB of 306.0 GiB) in 40m 16s, read: 494.3 MiB/s, write: 1.7 MiB/s
INFO:  47% (143.9 GiB of 306.0 GiB) in 40m 22s, read: 476.7 MiB/s, write: 1.3 MiB/s
INFO:  48% (147.3 GiB of 306.0 GiB) in 40m 28s, read: 570.0 MiB/s, write: 682.7 KiB/s
INFO:  49% (150.2 GiB of 306.0 GiB) in 40m 34s, read: 494.7 MiB/s, write: 2.0 MiB/s
INFO:  50% (153.2 GiB of 306.0 GiB) in 40m 41s, read: 449.7 MiB/s, write: 1.7 MiB/s
INFO:  51% (156.3 GiB of 306.0 GiB) in 40m 48s, read: 456.6 MiB/s, write: 2.3 MiB/s
INFO:  52% (159.4 GiB of 306.0 GiB) in 40m 55s, read: 449.1 MiB/s, write: 1.7 MiB/s
INFO:  53% (162.5 GiB of 306.0 GiB) in 41m 2s, read: 450.9 MiB/s, write: 1.1 MiB/s
INFO:  54% (165.4 GiB of 306.0 GiB) in 41m 8s, read: 496.0 MiB/s, write: 682.7 KiB/s
INFO:  55% (168.6 GiB of 306.0 GiB) in 41m 14s, read: 546.0 MiB/s, write: 22.7 MiB/s
INFO:  56% (171.4 GiB of 306.0 GiB) in 41m 24s, read: 284.8 MiB/s, write: 53.6 MiB/s
INFO:  57% (174.5 GiB of 306.0 GiB) in 41m 31s, read: 449.7 MiB/s, write: 2.3 MiB/s
INFO:  58% (177.6 GiB of 306.0 GiB) in 41m 38s, read: 452.0 MiB/s, write: 1.1 MiB/s
INFO:  59% (180.6 GiB of 306.0 GiB) in 41m 45s, read: 447.4 MiB/s, write: 585.1 KiB/s
INFO:  60% (183.7 GiB of 306.0 GiB) in 41m 52s, read: 450.3 MiB/s, write: 585.1 KiB/s
INFO:  61% (186.7 GiB of 306.0 GiB) in 41m 59s, read: 446.3 MiB/s, write: 5.7 MiB/s
INFO:  62% (189.8 GiB of 306.0 GiB) in 42m 6s, read: 445.1 MiB/s, write: 4.6 MiB/s
INFO:  63% (192.8 GiB of 306.0 GiB) in 42m 13s, read: 443.4 MiB/s, write: 4.6 MiB/s
INFO:  64% (196.1 GiB of 306.0 GiB) in 42m 20s, read: 473.7 MiB/s, write: 2.3 MiB/s
INFO:  65% (199.1 GiB of 306.0 GiB) in 42m 27s, read: 447.4 MiB/s, write: 1.7 MiB/s
INFO:  66% (202.2 GiB of 306.0 GiB) in 42m 34s, read: 450.3 MiB/s, write: 4.6 MiB/s
INFO:  67% (205.2 GiB of 306.0 GiB) in 42m 41s, read: 445.7 MiB/s, write: 8.0 MiB/s
INFO:  68% (208.4 GiB of 306.0 GiB) in 42m 48s, read: 461.7 MiB/s, write: 585.1 KiB/s
INFO:  69% (211.4 GiB of 306.0 GiB) in 42m 54s, read: 510.7 MiB/s, write: 8.0 MiB/s
INFO:  70% (214.6 GiB of 306.0 GiB) in 43m 1s, read: 472.6 MiB/s, write: 4.6 MiB/s
INFO:  71% (217.6 GiB of 306.0 GiB) in 43m 8s, read: 444.0 MiB/s, write: 8.0 MiB/s
INFO:  72% (220.7 GiB of 306.0 GiB) in 43m 15s, read: 448.0 MiB/s, write: 5.1 MiB/s
INFO:  73% (223.8 GiB of 306.0 GiB) in 43m 22s, read: 445.7 MiB/s, write: 5.1 MiB/s
INFO:  74% (228.2 GiB of 306.0 GiB) in 43m 27s, read: 903.2 MiB/s, write: 20.0 MiB/s
INFO:  76% (233.3 GiB of 306.0 GiB) in 43m 30s, read: 1.7 GiB/s, write: 22.7 MiB/s
INFO:  78% (240.5 GiB of 306.0 GiB) in 43m 33s, read: 2.4 GiB/s, write: 5.3 MiB/s
INFO:  81% (250.4 GiB of 306.0 GiB) in 43m 36s, read: 3.3 GiB/s, write: 0 B/s
INFO:  83% (255.0 GiB of 306.0 GiB) in 43m 39s, read: 1.5 GiB/s, write: 22.7 MiB/s
INFO:  84% (259.1 GiB of 306.0 GiB) in 43m 42s, read: 1.3 GiB/s, write: 24.0 MiB/s
INFO:  88% (269.3 GiB of 306.0 GiB) in 43m 45s, read: 3.4 GiB/s, write: 0 B/s
INFO:  91% (279.4 GiB of 306.0 GiB) in 43m 48s, read: 3.3 GiB/s, write: 0 B/s
INFO:  94% (289.6 GiB of 306.0 GiB) in 43m 51s, read: 3.4 GiB/s, write: 0 B/s
INFO:  97% (299.8 GiB of 306.0 GiB) in 43m 54s, read: 3.4 GiB/s, write: 0 B/s
INFO: 100% (306.0 GiB of 306.0 GiB) in 43m 57s, read: 2.1 GiB/s, write: 176.0 KiB/s
INFO: Waiting for server to finish backup validation...
INFO: backup is sparse: 107.19 GiB (35%) total zero data
INFO: backup was done incrementally, reused 276.47 GiB (90%)
INFO: transferred 306.00 GiB in 3877 seconds (80.8 MiB/s)
INFO: stopping kvm after backup task
VM quit/powerdown failed - terminating now with SIGTERM
INFO: adding notes to backup
INFO: Finished Backup of VM 108 (01:06:35)
INFO: Backup finished at 2025-01-26 09:07:25
INFO: Backup job finished successfully
TASK OK

proxmox-ve: 8.3.0 (running kernel: 6.8.12-7-pve)
pve-manager: 8.3.3 (running version: 8.3.3/f157a38b211595d6)
proxmox-kernel-helper: 8.1.0
proxmox-kernel-6.8: 6.8.12-7
proxmox-kernel-6.8.12-7-pve-signed: 6.8.12-7
proxmox-kernel-6.8.12-5-pve-signed: 6.8.12-5
proxmox-kernel-6.8.12-4-pve-signed: 6.8.12-4
ceph-fuse: 17.2.7-pve3
corosync: 3.1.7-pve3
criu: 3.17.1-2+deb12u1
glusterfs-client: 10.3-5
ifupdown2: 3.2.0-1+pmx11
intel-microcode: 3.20241112.1
ksm-control-daemon: 1.5-1
libjs-extjs: 7.0.0-5
libknet1: 1.28-pve1
libproxmox-acme-perl: 1.5.1
libproxmox-backup-qemu0: 1.4.1
libproxmox-rs-perl: 0.3.4
libpve-access-control: 8.2.0
libpve-apiclient-perl: 3.3.2
libpve-cluster-api-perl: 8.0.10
libpve-cluster-perl: 8.0.10
libpve-common-perl: 8.2.9
libpve-guest-common-perl: 5.1.6
libpve-http-server-perl: 5.1.2
libpve-network-perl: 0.10.0
libpve-rs-perl: 0.9.1
libpve-storage-perl: 8.3.3
libspice-server1: 0.15.1-1
lvm2: 2.03.16-2
lxc-pve: 6.0.0-1
lxcfs: 6.0.0-pve2
novnc-pve: 1.5.0-1
proxmox-backup-client: 3.3.2-1
proxmox-backup-file-restore: 3.3.2-2
proxmox-firewall: 0.6.0
proxmox-kernel-helper: 8.1.0
proxmox-mail-forward: 0.3.1
proxmox-mini-journalreader: 1.4.0
proxmox-offline-mirror-helper: 0.6.7
proxmox-widget-toolkit: 4.3.4
pve-cluster: 8.0.10
pve-container: 5.2.3
pve-docs: 8.3.1
pve-edk2-firmware: 4.2023.08-4
pve-esxi-import-tools: 0.7.2
pve-firewall: 5.1.0
pve-firmware: 3.14-3
pve-ha-manager: 4.0.6
pve-i18n: 3.3.3
pve-qemu-kvm: 9.0.2-4
pve-xtermjs: 5.3.0-3
qemu-server: 8.3.6
smartmontools: 7.3-pve1
spiceterm: 3.3.0
swtpm: 0.8.0+pve1
vncterm: 1.8.0
zfsutils-linux: 2.2.7-pve1

proxmox-backup: 3.3.0 (running kernel: 6.8.12-6-pve)
proxmox-backup-server: 3.3.2-1 (running version: 3.3.2)
proxmox-kernel-helper: 8.1.0
proxmox-kernel-6.8: 6.8.12-6
proxmox-kernel-6.8.12-6-pve-signed: 6.8.12-6
proxmox-kernel-6.8.12-5-pve-signed: 6.8.12-5
proxmox-kernel-6.8.12-4-pve-signed: 6.8.12-4
ifupdown2: 3.2.0-1+pmx11
libjs-extjs: 7.0.0-5
proxmox-backup-docs: 3.3.2-1
proxmox-backup-client: 3.3.2-1
proxmox-mail-forward: 0.3.1
proxmox-mini-journalreader: 1.4.0
proxmox-offline-mirror-helper: 0.6.7
proxmox-widget-toolkit: 4.3.4
pve-xtermjs: 5.3.0-3
smartmontools: 7.3-pve1
zfsutils-linux: 2.2.6-pve1

 

[Max Carrara]

Hey!

From what I can tell, the issue is most likely because you're using an HDD-backed storage. As mentioned in our server system requirements, we recommend using a storage that can deal well with random IO workloads. Regarding ZFS specifically, having a pool consisting of mirror vdevs will be better than having a pool of RAID-Z vdevs; conversely, a pool of RAID-Z vdevs will usually outperform a pool of mirrors when it comes to bandwidth (source).

In other words, running PBS in a VM on a consumer-grade NAS with HDDs and a ZFS configuration that favours bandwidth over IOPS is... yeah it's not going to be good for performance in that regard. PBS relies on random read/write performance for a lot of its functionality (due to the nature of algorithms etc. used).

You might want to consider getting a second of machine specifically for backups only; preferably one with SSDs that have power loss protection. (Why power loss protection? Because it's usually only quality SSDs that have that; it's a good heuristic if you want to avoid accidentally buying trash.)

Otherwise, it's in my opinion fine for a backup to take some time if you're just running things at home (and not in a company or something), so you could schedule your backups to only run during the night when the NAS isn't being used much. Though of course, that depends on your use case and preferences etc.

Hope that helps!

 

[onlyzeros]

Hi Max,
Thanks for you reply.
I only use the F4-424 for backups, from Truenas and from Proxmox.
Unfortunately I cannot have another system much less with ssd's, too expensive for me.

I guess I will just have to live with it.

Thanks anyway.