Backup Performance Issue

[saklikoy]

The read and write rates in Proxmox Backup Server backup are very low, so I think backups are progressing very slowly.

I tried many things but I couldn't find a solution. As far as I can see, I realized that this is related to compression.
I have done some tests and attached the results. It seems that when I turn the compression off, there is no problem, but as you know, compression is always on in PBS. Has anyone had this problem before and provide a solution?

Proxmox VE version is : 7.4-13
PBS version is : 2.4.-2

 

[alex#]

"Very slow" depends on many factors but your post doesn't provide more information about hardware/software configuration. Since the average rate of 135.9 MiB/s doesn't seem reasonable for you, can you also provide stats with the aforesaid compression turned off?

What did you try so far? Maybe take also a look at these settings: https://forum.proxmox.com/threads/backups-getting-hung-with-high-i-o-wait.128919/post-565281

 

[saklikoy]

Hello alex,

It is as follows.
Also, what I tried are the ones you recommended in the post you sent.Changing the number of max-workers.Changing the backup storage using SSD storage and tried with physical PBS.

INFO: starting new backup job: vzdump 120 --notes-template '{{guestname}}' --storage DDBackups --node xxx --compress 0 --mode snapshot --remove 0
INFO: Starting Backup of VM 120 (qemu)
INFO: Backup started at 2023-06-19 10:59:17
INFO: status = running
INFO: VM Name: xxxx
INFO: include disk 'virtio0' 'HW2PRXKKBDS03:120/vm-120-disk-0.qcow2' 100G
INFO: include disk 'virtio1' 'HW2PRXKKBDS03:120/vm-120-disk-1.qcow2' 300G
INFO: include disk 'virtio2' 'PRXKKBDS01:120/vm-120-disk-3.qcow2' 200G
INFO: include disk 'efidisk0' 'HW2PRXKKBDS03:120/vm-120-disk-2.qcow2' 528K
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: skip unused drive 'PRXKKBDS01:120/vm-120-disk-0.qcow2' (not included into backup)
INFO: creating vzdump archive '/mnt/pve/DDBackups/dump/vzdump-qemu-120-2023_06_19-10_59_17.vma'
INFO: issuing guest-agent 'fs-freeze' command
INFO: issuing guest-agent 'fs-thaw' command
INFO: started backup task '6b28f43a-d135-497b-a7e3-49647f47a4e4'
INFO: resuming VM again
INFO: 0% (1.7 GiB of 600.0 GiB) in 3s, read: 589.7 MiB/s, write: 572.1 MiB/s
INFO: 1% (7.4 GiB of 600.0 GiB) in 12s, read: 648.1 MiB/s, write: 604.6 MiB/s
INFO: 2% (12.8 GiB of 600.0 GiB) in 17s, read: 1.1 GiB/s, write: 529.4 MiB/s
INFO: 3% (20.2 GiB of 600.0 GiB) in 24s, read: 1.1 GiB/s, write: 82.3 KiB/s
INFO: 4% (25.3 GiB of 600.0 GiB) in 27s, read: 1.7 GiB/s, write: 0 B/s
INFO: 5% (33.2 GiB of 600.0 GiB) in 30s, read: 2.6 GiB/s, write: 85.3 KiB/s
INFO: 6% (37.8 GiB of 600.0 GiB) in 36s, read: 781.8 MiB/s, write: 559.7 MiB/s
INFO: 7% (44.0 GiB of 600.0 GiB) in 45s, read: 711.3 MiB/s, write: 711.3 MiB/s
INFO: 8% (49.0 GiB of 600.0 GiB) in 52s, read: 735.4 MiB/s, write: 735.3 MiB/s
INFO: 9% (54.6 GiB of 600.0 GiB) in 1m 3s, read: 521.0 MiB/s, write: 521.0 MiB/s
INFO: 10% (60.6 GiB of 600.0 GiB) in 1m 39s, read: 168.8 MiB/s, write: 131.9 MiB/s
INFO: 11% (66.2 GiB of 600.0 GiB) in 1m 49s, read: 577.0 MiB/s, write: 227.2 MiB/s
INFO: 12% (72.4 GiB of 600.0 GiB) in 2m 2s, read: 492.0 MiB/s, write: 477.5 MiB/s
INFO: 13% (78.5 GiB of 600.0 GiB) in 2m 16s, read: 440.6 MiB/s, write: 117.8 MiB/s
INFO: 14% (84.3 GiB of 600.0 GiB) in 2m 22s, read: 997.7 MiB/s, write: 1010.7 KiB/s
INFO: 15% (90.4 GiB of 600.0 GiB) in 2m 28s, read: 1.0 GiB/s, write: 1.5 MiB/s
INFO: 16% (96.7 GiB of 600.0 GiB) in 2m 34s, read: 1.1 GiB/s, write: 1.1 MiB/s
INFO: 17% (103.2 GiB of 600.0 GiB) in 2m 40s, read: 1.1 GiB/s, write: 1.1 MiB/s
INFO: 18% (108.3 GiB of 600.0 GiB) in 2m 45s, read: 1.0 GiB/s, write: 1.4 MiB/s
INFO: 19% (114.6 GiB of 600.0 GiB) in 2m 51s, read: 1.0 GiB/s, write: 1.3 MiB/s
INFO: 20% (120.7 GiB of 600.0 GiB) in 2m 57s, read: 1.0 GiB/s, write: 1.0 MiB/s
INFO: 21% (126.1 GiB of 600.0 GiB) in 3m 2s, read: 1.1 GiB/s, write: 1.1 MiB/s
INFO: 22% (132.8 GiB of 600.0 GiB) in 3m 10s, read: 858.8 MiB/s, write: 509.3 MiB/s
INFO: 23% (138.8 GiB of 600.0 GiB) in 3m 16s, read: 1011.3 MiB/s, write: 10.7 KiB/s
INFO: 24% (144.7 GiB of 600.0 GiB) in 3m 28s, read: 503.8 MiB/s, write: 0 B/s
INFO: 25% (150.2 GiB of 600.0 GiB) in 3m 35s, read: 804.9 MiB/s, write: 18.3 KiB/s
INFO: 26% (156.5 GiB of 600.0 GiB) in 3m 42s, read: 932.3 MiB/s, write: 0 B/s
INFO: 27% (162.7 GiB of 600.0 GiB) in 3m 49s, read: 898.5 MiB/s, write: 36.6 KiB/s
INFO: 28% (168.8 GiB of 600.0 GiB) in 3m 55s, read: 1.0 GiB/s, write: 10.7 KiB/s
INFO: 29% (174.1 GiB of 600.0 GiB) in 4m, read: 1.1 GiB/s, write: 25.6 KiB/s
INFO: 30% (180.9 GiB of 600.0 GiB) in 4m 7s, read: 986.8 MiB/s, write: 66.9 KiB/s
INFO: 31% (187.0 GiB of 600.0 GiB) in 4m 13s, read: 1.0 GiB/s, write: 0 B/s
INFO: 32% (192.2 GiB of 600.0 GiB) in 4m 19s, read: 884.7 MiB/s, write: 0 B/s
INFO: 33% (198.7 GiB of 600.0 GiB) in 4m 25s, read: 1.1 GiB/s, write: 32.0 KiB/s
INFO: 34% (204.5 GiB of 600.0 GiB) in 4m 31s, read: 992.6 MiB/s, write: 746.8 MiB/s
INFO: 35% (210.9 GiB of 600.0 GiB) in 4m 40s, read: 727.9 MiB/s, write: 727.9 MiB/s
INFO: 36% (216.8 GiB of 600.0 GiB) in 4m 48s, read: 763.3 MiB/s, write: 763.2 MiB/s
INFO: 37% (222.2 GiB of 600.0 GiB) in 4m 57s, read: 612.6 MiB/s, write: 608.6 MiB/s
INFO: 38% (228.7 GiB of 600.0 GiB) in 5m 8s, read: 605.3 MiB/s, write: 605.3 MiB/s
INFO: 39% (234.3 GiB of 600.0 GiB) in 5m 17s, read: 639.1 MiB/s, write: 639.1 MiB/s
INFO: 40% (240.1 GiB of 600.0 GiB) in 5m 27s, read: 594.9 MiB/s, write: 594.8 MiB/s
INFO: 41% (246.1 GiB of 600.0 GiB) in 5m 38s, read: 558.9 MiB/s, write: 558.9 MiB/s
INFO: 42% (252.0 GiB of 600.0 GiB) in 5m 48s, read: 603.0 MiB/s, write: 603.0 MiB/s
INFO: 43% (258.2 GiB of 600.0 GiB) in 5m 58s, read: 636.0 MiB/s, write: 636.0 MiB/s
INFO: 44% (264.6 GiB of 600.0 GiB) in 6m 7s, read: 717.4 MiB/s, write: 717.4 MiB/s
INFO: 45% (270.3 GiB of 600.0 GiB) in 6m 17s, read: 593.3 MiB/s, write: 593.3 MiB/s
INFO: 46% (276.1 GiB of 600.0 GiB) in 6m 25s, read: 729.9 MiB/s, write: 729.9 MiB/s
INFO: 47% (283.0 GiB of 600.0 GiB) in 6m 34s, read: 793.7 MiB/s, write: 793.6 MiB/s
INFO: 48% (288.0 GiB of 600.0 GiB) in 6m 41s, read: 729.1 MiB/s, write: 728.5 MiB/s
INFO: 49% (294.3 GiB of 600.0 GiB) in 6m 50s, read: 710.8 MiB/s, write: 710.8 MiB/s
INFO: 50% (300.1 GiB of 600.0 GiB) in 6m 57s, read: 854.6 MiB/s, write: 851.5 MiB/s
INFO: 51% (310.8 GiB of 600.0 GiB) in 7m 4s, read: 1.5 GiB/s, write: 567.8 MiB/s
INFO: 52% (313.0 GiB of 600.0 GiB) in 7m 7s, read: 746.4 MiB/s, write: 640.6 MiB/s
INFO: 53% (318.4 GiB of 600.0 GiB) in 7m 13s, read: 921.0 MiB/s, write: 762.4 MiB/s
INFO: 54% (324.4 GiB of 600.0 GiB) in 7m 20s, read: 875.2 MiB/s, write: 733.1 MiB/s
INFO: 55% (330.5 GiB of 600.0 GiB) in 7m 42s, read: 284.8 MiB/s, write: 25.5 MiB/s
INFO: 56% (336.1 GiB of 600.0 GiB) in 7m 47s, read: 1.1 GiB/s, write: 1.1 MiB/s
INFO: 57% (343.1 GiB of 600.0 GiB) in 7m 53s, read: 1.2 GiB/s, write: 1.2 MiB/s
INFO: 58% (348.7 GiB of 600.0 GiB) in 7m 58s, read: 1.1 GiB/s, write: 1.1 MiB/s
INFO: 59% (354.4 GiB of 600.0 GiB) in 8m 3s, read: 1.1 GiB/s, write: 1.2 MiB/s
INFO: 60% (360.1 GiB of 600.0 GiB) in 8m 8s, read: 1.1 GiB/s, write: 1.2 MiB/s
INFO: 61% (366.8 GiB of 600.0 GiB) in 8m 14s, read: 1.1 GiB/s, write: 1.1 MiB/s
INFO: 62% (372.5 GiB of 600.0 GiB) in 8m 19s, read: 1.1 GiB/s, write: 1.2 MiB/s
INFO: 63% (378.3 GiB of 600.0 GiB) in 8m 24s, read: 1.1 GiB/s, write: 1.1 MiB/s
INFO: 64% (385.2 GiB of 600.0 GiB) in 8m 30s, read: 1.2 GiB/s, write: 1.2 MiB/s
INFO: 65% (391.2 GiB of 600.0 GiB) in 8m 36s, read: 1.0 GiB/s, write: 87.2 MiB/s
INFO: 66% (399.1 GiB of 600.0 GiB) in 8m 40s, read: 2.0 GiB/s, write: 312.4 MiB/s
INFO: 67% (403.7 GiB of 600.0 GiB) in 8m 43s, read: 1.6 GiB/s, write: 89.3 MiB/s
INFO: 69% (417.8 GiB of 600.0 GiB) in 8m 46s, read: 4.7 GiB/s, write: 232.8 MiB/s
INFO: 71% (427.9 GiB of 600.0 GiB) in 8m 49s, read: 3.4 GiB/s, write: 3.4 MiB/s
INFO: 72% (432.4 GiB of 600.0 GiB) in 8m 53s, read: 1.1 GiB/s, write: 1.1 MiB/s
INFO: 73% (438.5 GiB of 600.0 GiB) in 8m 59s, read: 1.0 GiB/s, write: 1.0 MiB/s
INFO: 74% (444.2 GiB of 600.0 GiB) in 9m 4s, read: 1.1 GiB/s, write: 1.2 MiB/s
INFO: 75% (450.9 GiB of 600.0 GiB) in 9m 10s, read: 1.1 GiB/s, write: 1.1 MiB/s
INFO: 76% (456.7 GiB of 600.0 GiB) in 9m 15s, read: 1.1 GiB/s, write: 1.1 MiB/s
INFO: 77% (462.4 GiB of 600.0 GiB) in 9m 20s, read: 1.2 GiB/s, write: 1.2 MiB/s
INFO: 78% (468.0 GiB of 600.0 GiB) in 9m 25s, read: 1.1 GiB/s, write: 1.2 MiB/s
INFO: 79% (474.9 GiB of 600.0 GiB) in 9m 31s, read: 1.1 GiB/s, write: 1.2 MiB/s
INFO: 80% (480.6 GiB of 600.0 GiB) in 9m 36s, read: 1.1 GiB/s, write: 1.1 MiB/s
INFO: 81% (486.2 GiB of 600.0 GiB) in 9m 41s, read: 1.1 GiB/s, write: 1.1 MiB/s
INFO: 82% (492.9 GiB of 600.0 GiB) in 9m 47s, read: 1.1 GiB/s, write: 1.1 MiB/s
INFO: 83% (498.7 GiB of 600.0 GiB) in 9m 52s, read: 1.1 GiB/s, write: 354.4 KiB/s
INFO: 84% (505.4 GiB of 600.0 GiB) in 10m, read: 862.0 MiB/s, write: 653.2 MiB/s
INFO: 85% (511.1 GiB of 600.0 GiB) in 10m 8s, read: 733.0 MiB/s, write: 718.3 MiB/s
INFO: 86% (516.8 GiB of 600.0 GiB) in 10m 15s, read: 826.0 MiB/s, write: 815.2 MiB/s
INFO: 87% (523.4 GiB of 600.0 GiB) in 10m 24s, read: 760.2 MiB/s, write: 741.7 MiB/s
INFO: 88% (528.3 GiB of 600.0 GiB) in 10m 45s, read: 236.3 MiB/s, write: 184.0 MiB/s
INFO: 89% (534.9 GiB of 600.0 GiB) in 10m 52s, read: 969.3 MiB/s, write: 952.6 MiB/s
INFO: 90% (541.0 GiB of 600.0 GiB) in 11m, read: 778.3 MiB/s, write: 773.6 MiB/s
INFO: 91% (547.3 GiB of 600.0 GiB) in 11m 10s, read: 647.9 MiB/s, write: 647.0 MiB/s
INFO: 92% (552.8 GiB of 600.0 GiB) in 11m 17s, read: 803.2 MiB/s, write: 800.0 MiB/s
INFO: 93% (559.1 GiB of 600.0 GiB) in 11m 26s, read: 719.4 MiB/s, write: 716.7 MiB/s
INFO: 94% (564.7 GiB of 600.0 GiB) in 11m 35s, read: 636.4 MiB/s, write: 634.6 MiB/s
INFO: 95% (570.9 GiB of 600.0 GiB) in 11m 44s, read: 704.1 MiB/s, write: 701.3 MiB/s
INFO: 96% (576.4 GiB of 600.0 GiB) in 12m 15s, read: 180.1 MiB/s, write: 0 B/s
INFO: 97% (582.8 GiB of 600.0 GiB) in 12m 23s, read: 825.0 MiB/s, write: 570.7 MiB/s
INFO: 98% (588.1 GiB of 600.0 GiB) in 12m 29s, read: 893.1 MiB/s, write: 0 B/s
INFO: 99% (594.3 GiB of 600.0 GiB) in 12m 35s, read: 1.0 GiB/s, write: 0 B/s
INFO: 100% (600.0 GiB of 600.0 GiB) in 12m 40s, read: 1.1 GiB/s, write: 99.2 KiB/s
INFO: backup is sparse: 359.75 GiB (59%) total zero data
INFO: transferred 600.00 GiB in 760 seconds (808.4 MiB/s)
INFO: archive file size: 240.33GB
INFO: adding notes to backup
INFO: Finished Backup of VM 120 (00:12:44)
INFO: Backup finished at 2023-06-19 11:12:01
INFO: Backup job finished successfully
TASK OK

 

[alex#]

A "physical PBS" is how it's designed for, in combination with an SSD and a good network connection you can expect quite fast backups.

What's the storage type of 'DDBackups' and how is it connected?

pvesm status

How is your PBS connected and how does the host system of your PBS looks like?

 

[_gabriel]

pbs is blazing fast on next backup thanks to dirty-map. so only changed blocks will be transfered.
first backup to pbs is limited to your 1GB/s network.
moreover if backup is done after a shutdown/start vm, whole vm disk need to be read at speed of the vm disk, but only changed blocks will be transfered to pbs.

 

[saklikoy]

A "physical PBS" is how it's designed for, in combination with an SSD and a good network connection you can expect quite fast backups.

I tried this but the result was not much different. I tried it in 10G network environment and with SSD disk. PBS is currently running as a virtual machine on Proxmox VE.

What's the storage type of 'DDBackups' and how is it connected?

It's a Dell Data Domain. it's connected via NFS
Name Type Status Total Used Available %
DDBackups nfs active 1276618733567 843061040128 433557693440 66.04%
DRTest nfs active 1073741824 17929472 1055812352 1.67%
HW2PRXKKBDS01 nfs active 10737418240 8036924928 2700493312 74.85%
HW2PRXKKBDS02 nfs active 5368709120 838362368 4530346752 15.62%
HW2PRXKKBDS03 nfs active 5368709120 1560937472 3807771648 29.07%
HW2PRXKKBDS04 nfs active 10737418240 6803688448 3933729792 63.36%
HW2PRXKKBDS05 nfs active 5368709120 727177728 4641531392 13.54%
HW2PRXKKBDS06 nfs active 5368709120 2530469888 2838239232 47.13%
HW2PRXKKBDS07 nfs active 5368709120 745128704 4623580416 13.88%
HW2PRXKKBDS08 nfs active 5368709120 781651968 4587057152 14.56%
HW2PRXKKBDS09 nfs active 5368709120 965781248 4402927872 17.99%
ISO-Domain nfs active 1276618733567 843060302848 433558430720 66.04%
PBS pbs active 1276618733568 843060363264 433558370304 66.04%
PRXKKBDS01 nfs active 42949672960 19160901632 23788771328 44.61%
PRXKKBDS02 nfs active 32212254720 17431745792 14780508928 54.12%
PRXKKBDS03 nfs active 21474836480 12841353984 8633482496 59.80%

How is your PBS connected and how does the host system of your PBS looks like?

I didn't understand what you mean.

 

[saklikoy]

pbs is blazing fast on next backup thanks to dirty-map. so only changed blocks will be transfered.
first backup to pbs is limited to your 1GB/s network.
moreover if backup is done after a shutdown/start vm, whole vm disk need to be read at speed of the vm disk, but only changed blocks will be transfered to pbs.

I absolutely agree with this, but I do not take very fast backups, yes, there is a lot of data changing, but I see a backup speed between 10-100Mb / s. In the built backup, if I turn off the compression, I can see that it is very fast as I said before.
I also shared the backup job output above.

 

[_gabriel]

the LOG from first post is PBS one (better if content is in a code Tag please, too difficult to read and re read) , it's slow because first backup since boot of the VM (log say dirty-bitmap status: created new)
if you start another backup few minutes later or next day, it will be fast, because not full vm disk need to be read.

the second LOG (btw please edit post to put LOG content inside a code tag, it's too difficult to read), is a log of a vzdump backup.

oh, I understand while writing, your local disk vm speed seems speedy but not with pbs.
What's cpu on pve and on PBS ?

edit: indeed, pbs is a other host ? a vm ? because seems capped to 1 GB network.

 

[saklikoy]

the LOG from first post is PBS one (better if content is in a code Tag please, too difficult to read and re read) , it's slow because first backup since boot of the VM (log say dirty-bitmap status: created new)
if you start another backup few minutes later or next day, it will be fast, because not full vm disk need to be read.

the second LOG (btw please edit post to put LOG content inside a code tag, it's too difficult to read), is a log of a vzdump backup.

oh, I understand while writing, your local disk vm speed seems speedy but not with pbs.
What's cpu on pve and on PBS ?

edit: indeed, pbs is a other host ? a vm ? because seems capped to 1 GB network.

PVE cpu : 192 x Intel(R) Xeon(R) Gold 6330H CPU @ 2.00GHz (4 Sockets)
PBS cpu : 4 socket 12 core = 48 vcpu
PVE network : 2x10G = active active LACP
PBS network : virtio 10G network

 

[_gabriel]

vcpu type of pbs vm is set to host ?

 

[saklikoy]

vcpu type of pbs vm is set to host ?

Yes

 

[_gabriel]

what about pbs benchmark ?
what about iperf between host and VM ?
You can try compare with pbs installed alongside the PVE, even if it's not supported, it works great here