Backup slow / froze VM / Upload speed

A

andaga

Member
Proxmox Subscriber
May 24, 2022
136
17
23
Switzerland
Hello,

I am facing an issue with backups on my Proxmox backup server with PBS. Please check the below details regarding my backup setup

Proxmox hosts pve1 and pve2

pve1 10.0.1.11
pve2 10.0.1.12

backup server 10.0.1.10

The datastore is created with Synology Storage mounted as NFS in the backup server. All the disk in the Synology Storage is upgraded to SSD now.

The VMs froze while the PBS backup is running on proxmox. The proxmox hosts have an AMD Epyc 7402Performance processor.

We have performed benchmark on Proxmox hosts and backup server with the datastore and noticed the below results. The upload speed on both Proxmox host is very slow.

pve2
┌───────────────────────────────────┬────────────────────┐
│ Name │ Value │
╞═══════════════════════════════════╪════════════════════╡
│ TLS (maximal backup upload speed) │ 117.57 MB/s (10%) │
├───────────────────────────────────┼────────────────────┤
│ SHA256 checksum computation speed │ 892.81 MB/s (44%) │
├───────────────────────────────────┼────────────────────┤
│ ZStd level 1 compression speed │ 495.09 MB/s (66%) │
├───────────────────────────────────┼────────────────────┤
│ ZStd level 1 decompression speed │ 833.13 MB/s (70%) │
├───────────────────────────────────┼────────────────────┤
│ Chunk verification speed │ 552.23 MB/s (73%) │
├───────────────────────────────────┼────────────────────┤
│ AES256 GCM encryption speed │ 2014.78 MB/s (55%) │
└───────────────────────────────────┴────────────────────┘
pve1
┌───────────────────────────────────┬────────────────────┐
│ Name │ Value │
╞═══════════════════════════════════╪════════════════════╡
│ TLS (maximal backup upload speed) │ 112.46 MB/s (9%) │
├───────────────────────────────────┼────────────────────┤
│ SHA256 checksum computation speed │ 889.39 MB/s (44%) │
├───────────────────────────────────┼────────────────────┤
│ ZStd level 1 compression speed │ 486.53 MB/s (65%) │
├───────────────────────────────────┼────────────────────┤
│ ZStd level 1 decompression speed │ 831.46 MB/s (69%) │
├───────────────────────────────────┼────────────────────┤
│ Chunk verification speed │ 553.74 MB/s (73%) │
├───────────────────────────────────┼────────────────────┤
│ AES256 GCM encryption speed │ 1978.83 MB/s (54%) │
└───────────────────────────────────┴────────────────────┘
Backup server
┌──────────────────────────────────────────────────────────
│ Name │ Value │
╞═══════════════════════════════════╪═════════════════════╡
│ TLS (maximal backup upload speed) │ 1484.15 MB/s (120%) │
├──────────────────────────────────────────────────────────
│ SHA256 checksum computation speed │ 619.24 MB/s (31%) │
├──────────────────────────────────────────────────────────
│ ZStd level 1 compression speed │ 698.54 MB/s (93%) │
├──────────────────────────────────────────────────────────
│ ZStd level 1 decompression speed │ 992.81 MB/s (83%) │
├──────────────────────────────────────────────────────────
│ Chunk verification speed │ 381.30 MB/s (50%) │
├──────────────────────────────────────────────────────────
│ AES256 GCM encryption speed │ 2180.47 MB/s (60%) │
Also, the write speed is not showing correctly.
```
INFO: 66% (99.0 GiB of 150.0 GiB) in 15m 34s, read: 99.2 MiB/s, write: 99.2 MiB/s
INFO: 67% (100.6 GiB of 150.0 GiB) in 15m 48s, read: 115.1 MiB/s, write: 115.1 MiB/s
INFO: 68% (102.0 GiB of 150.0 GiB) in 16m 2s, read: 104.6 MiB/s, write: 48.6 MiB/s
INFO: 69% (103.5 GiB of 150.0 GiB) in 16m 16s, read: 112.3 MiB/s, write: 0 B/s
INFO: 70% (105.1 GiB of 150.0 GiB) in 16m 30s, read: 112.6 MiB/s, write: 0 B/s
INFO: 71% (106.5 GiB of 150.0 GiB) in 16m 43s, read: 112.3 MiB/s, write: 0 B/s
INFO: 72% (108.0 GiB of 150.0 GiB) in 16m 57s, read: 112.6 MiB/s, write: 0 B/s
INFO: 73% (109.6 GiB of 150.0 GiB) in 17m 11s, read: 112.3 MiB/s, write: 0 B/s
INFO: 74% (111.0 GiB of 150.0 GiB) in 17m 24s, read: 112.6 MiB/s, write: 28.9 MiB/s
INFO: 75% (112.5 GiB of 150.0 GiB) in 17m 38s, read: 112.3 MiB/s, write: 2.3 MiB/s

```


Could you please check and let us know if any bug present in proxmox causing the slow upload and Vm froze issue?
 
just to note: 112MiB/s is near the theoretical limit of 1Gbit/s, so you just may run into your network limit?
 
@dcsapak
I think so too.
BTW: iSCSI for PBS is much faster when it comes to GC or related (especially for Synology NAS because of its NFS limitation).
 
Last edited:
This is was we get on an Synology over iSCSI with spinning Disks:

Code: 
Uploaded 370 chunks in 5 seconds.
Time per request: 13724 microseconds.
TLS speed: 305.60 MB/s
SHA256 speed: 1978.42 MB/s
Compression speed: 816.53 MB/s
Decompress speed: 1091.22 MB/s
AES256/GCM speed: 2589.88 MB/s
Verify speed: 702.67 MB/s
┌───────────────────────────────────┬────────────────────┐
│ Name                              │ Value              │
╞═══════════════════════════════════╪════════════════════╡
│ TLS (maximal backup upload speed) │ 305.60 MB/s (25%)  │
├───────────────────────────────────┼────────────────────┤
│ SHA256 checksum computation speed │ 1978.42 MB/s (98%) │
├───────────────────────────────────┼────────────────────┤
│ ZStd level 1 compression speed    │ 816.53 MB/s (109%) │
├───────────────────────────────────┼────────────────────┤
│ ZStd level 1 decompression speed  │ 1091.22 MB/s (91%) │
├───────────────────────────────────┼────────────────────┤
│ Chunk verification speed          │ 702.67 MB/s (93%)  │
├───────────────────────────────────┼────────────────────┤
│ AES256 GCM encryption speed       │ 2589.88 MB/s (71%) │
└───────────────────────────────────┴────────────────────┘

Real Performance (Zeroes) is about 16-25 GBytes/second, otherwise 350-450 MBytes/second.
 
Last edited:
Hello! Thank you for your feedback! But each time if I use ProxMox Backup Server.... VM Freeze
If I use VZDUMP VM freeze!

I don't know what to do! With ESXI in the past, we have never this issue on the same hardware!

Someone have an idea?
 
andaga said:
Hello! Thank you for your feedback! But each time if I use ProxMox Backup Server.... VM Freeze
If I use VZDUMP VM freeze!
Click to expand...
what exactly do you mean with that? the way backup works is that if a write comes from the vm, the backup blocks that until it has backed up the current state of that block on disk (if that block was not already backed up)
 
can you post the task log of such a backup?
 
Task viewer: VM/CT 103 - Backup


OutputStatus

Stop
INFO: starting new backup job: vzdump 103 --node pve1 --notes-template '{{guestname}}' --mode snapshot --storage PMBackup2 --remove 0 --compress zstd
INFO: Starting Backup of VM 103 (qemu)
INFO: Backup started at 2022-05-29 17:42:20
INFO: status = running
INFO: VM Name: PleskTython
INFO: include disk 'scsi0' 'DCSAN:103/vm-103-disk-0.qcow2' 150G
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating vzdump archive '/mnt/pve/PMBackup2/dump/vzdump-qemu-103-2022_05_29-17_42_20.vma.zst'
INFO: issuing guest-agent 'fs-freeze' command
INFO: issuing guest-agent 'fs-thaw' command
INFO: started backup task 'd1cc8b26-467a-4750-a492-655c5398338c'
INFO: resuming VM again
INFO: 0% (348.0 MiB of 150.0 GiB) in 3s, read: 116.0 MiB/s, write: 108.4 MiB/s
INFO: 1% (1.6 GiB of 150.0 GiB) in 14s, read: 117.3 MiB/s, write: 44.7 MiB/s
INFO: 2% (3.0 GiB of 150.0 GiB) in 27s, read: 112.7 MiB/s, write: 29.4 MiB/s
INFO: 3% (4.6 GiB of 150.0 GiB) in 41s, read: 112.4 MiB/s, write: 128.3 KiB/s
INFO: 4% (6.1 GiB of 150.0 GiB) in 55s, read: 112.3 MiB/s, write: 72.2 MiB/s
INFO: 5% (7.5 GiB of 150.0 GiB) in 1m 8s, read: 111.6 MiB/s, write: 110.7 MiB/s
ERROR: interrupted by signal
INFO: aborting backup job
INFO: resuming VM again
ERROR: Backup of VM 103 failed - interrupted by signal
 
i mean it probably does not have to with your problem, but you don't backup to a pbs but a regular vzdump backup? (since the thread was about pbs)

as i said, during backup, the writes from inside the vm to blocks that were not backed up yet, must be paused (until they are backed up)
maybe the backup does saturate your network and the vm cannot access the disks anymore ?

can you post the vm config and storage config too ?
 
You must log in or register to reply here.
Top Bottom