Backup slow / froze VM / Upload speed

May 24, 2022
7
0
1
Hello,

I am facing an issue with backups on my Proxmox backup server with PBS. Please check the below details regarding my backup setup

Proxmox hosts pve1 and pve2

pve1 10.0.1.11
pve2 10.0.1.12

backup server 10.0.1.10

The datastore is created with Synology Storage mounted as NFS in the backup server. All the disk in the Synology Storage is upgraded to SSD now.

The VMs froze while the PBS backup is running on proxmox. The proxmox hosts have an AMD Epyc 7402Performance processor.

We have performed benchmark on Proxmox hosts and backup server with the datastore and noticed the below results. The upload speed on both Proxmox host is very slow.

pve2
┌───────────────────────────────────┬────────────────────┐
│ Name │ Value │
╞═══════════════════════════════════╪════════════════════╡
│ TLS (maximal backup upload speed) │ 117.57 MB/s (10%) │
├───────────────────────────────────┼────────────────────┤
│ SHA256 checksum computation speed │ 892.81 MB/s (44%) │
├───────────────────────────────────┼────────────────────┤
│ ZStd level 1 compression speed │ 495.09 MB/s (66%) │
├───────────────────────────────────┼────────────────────┤
│ ZStd level 1 decompression speed │ 833.13 MB/s (70%) │
├───────────────────────────────────┼────────────────────┤
│ Chunk verification speed │ 552.23 MB/s (73%) │
├───────────────────────────────────┼────────────────────┤
│ AES256 GCM encryption speed │ 2014.78 MB/s (55%) │
└───────────────────────────────────┴────────────────────┘
pve1
┌───────────────────────────────────┬────────────────────┐
│ Name │ Value │
╞═══════════════════════════════════╪════════════════════╡
│ TLS (maximal backup upload speed) │ 112.46 MB/s (9%) │
├───────────────────────────────────┼────────────────────┤
│ SHA256 checksum computation speed │ 889.39 MB/s (44%) │
├───────────────────────────────────┼────────────────────┤
│ ZStd level 1 compression speed │ 486.53 MB/s (65%) │
├───────────────────────────────────┼────────────────────┤
│ ZStd level 1 decompression speed │ 831.46 MB/s (69%) │
├───────────────────────────────────┼────────────────────┤
│ Chunk verification speed │ 553.74 MB/s (73%) │
├───────────────────────────────────┼────────────────────┤
│ AES256 GCM encryption speed │ 1978.83 MB/s (54%) │
└───────────────────────────────────┴────────────────────┘
Backup server
┌──────────────────────────────────────────────────────────
│ Name │ Value │
╞═══════════════════════════════════╪═════════════════════╡
│ TLS (maximal backup upload speed) │ 1484.15 MB/s (120%) │
├──────────────────────────────────────────────────────────
│ SHA256 checksum computation speed │ 619.24 MB/s (31%) │
├──────────────────────────────────────────────────────────
│ ZStd level 1 compression speed │ 698.54 MB/s (93%) │
├──────────────────────────────────────────────────────────
│ ZStd level 1 decompression speed │ 992.81 MB/s (83%) │
├──────────────────────────────────────────────────────────
│ Chunk verification speed │ 381.30 MB/s (50%) │
├──────────────────────────────────────────────────────────
│ AES256 GCM encryption speed │ 2180.47 MB/s (60%) │
Also, the write speed is not showing correctly.
```
INFO: 66% (99.0 GiB of 150.0 GiB) in 15m 34s, read: 99.2 MiB/s, write: 99.2 MiB/s
INFO: 67% (100.6 GiB of 150.0 GiB) in 15m 48s, read: 115.1 MiB/s, write: 115.1 MiB/s
INFO: 68% (102.0 GiB of 150.0 GiB) in 16m 2s, read: 104.6 MiB/s, write: 48.6 MiB/s
INFO: 69% (103.5 GiB of 150.0 GiB) in 16m 16s, read: 112.3 MiB/s, write: 0 B/s
INFO: 70% (105.1 GiB of 150.0 GiB) in 16m 30s, read: 112.6 MiB/s, write: 0 B/s
INFO: 71% (106.5 GiB of 150.0 GiB) in 16m 43s, read: 112.3 MiB/s, write: 0 B/s
INFO: 72% (108.0 GiB of 150.0 GiB) in 16m 57s, read: 112.6 MiB/s, write: 0 B/s
INFO: 73% (109.6 GiB of 150.0 GiB) in 17m 11s, read: 112.3 MiB/s, write: 0 B/s
INFO: 74% (111.0 GiB of 150.0 GiB) in 17m 24s, read: 112.6 MiB/s, write: 28.9 MiB/s
INFO: 75% (112.5 GiB of 150.0 GiB) in 17m 38s, read: 112.3 MiB/s, write: 2.3 MiB/s

```


Could you please check and let us know if any bug present in proxmox causing the slow upload and Vm froze issue?
 
May 24, 2022
7
0
1
Thank you @ITT for your response. We have try your thread but don't work! The benchmark still slow.

We are changing from NFS to ISCSI for trial. But I don't think is the issue.
 

dcsapak

Proxmox Staff Member
Staff member
Feb 1, 2016
7,874
952
163
33
Vienna
just to note: 112MiB/s is near the theoretical limit of 1Gbit/s, so you just may run into your network limit?
 

ITT

Active Member
Mar 19, 2021
188
46
28
41
@dcsapak
I think so too.
BTW: iSCSI for PBS is much faster when it comes to GC or related (especially for Synology NAS because of its NFS limitation).
 
Last edited:

ITT

Active Member
Mar 19, 2021
188
46
28
41
This is was we get on an Synology over iSCSI with spinning Disks:

Code:
Uploaded 370 chunks in 5 seconds.
Time per request: 13724 microseconds.
TLS speed: 305.60 MB/s
SHA256 speed: 1978.42 MB/s
Compression speed: 816.53 MB/s
Decompress speed: 1091.22 MB/s
AES256/GCM speed: 2589.88 MB/s
Verify speed: 702.67 MB/s
┌───────────────────────────────────┬────────────────────┐
│ Name                              │ Value              │
╞═══════════════════════════════════╪════════════════════╡
│ TLS (maximal backup upload speed) │ 305.60 MB/s (25%)  │
├───────────────────────────────────┼────────────────────┤
│ SHA256 checksum computation speed │ 1978.42 MB/s (98%) │
├───────────────────────────────────┼────────────────────┤
│ ZStd level 1 compression speed    │ 816.53 MB/s (109%) │
├───────────────────────────────────┼────────────────────┤
│ ZStd level 1 decompression speed  │ 1091.22 MB/s (91%) │
├───────────────────────────────────┼────────────────────┤
│ Chunk verification speed          │ 702.67 MB/s (93%)  │
├───────────────────────────────────┼────────────────────┤
│ AES256 GCM encryption speed       │ 2589.88 MB/s (71%) │
└───────────────────────────────────┴────────────────────┘

Real Performance (Zeroes) is about 16-25 GBytes/second, otherwise 350-450 MBytes/second.
 
Last edited:
May 24, 2022
7
0
1
Hello! Thank you for your feedback! But each time if I use ProxMox Backup Server.... VM Freeze
If I use VZDUMP VM freeze!

I don't know what to do! With ESXI in the past, we have never this issue on the same hardware!

Someone have an idea?
 

dcsapak

Proxmox Staff Member
Staff member
Feb 1, 2016
7,874
952
163
33
Vienna
Hello! Thank you for your feedback! But each time if I use ProxMox Backup Server.... VM Freeze
If I use VZDUMP VM freeze!
what exactly do you mean with that? the way backup works is that if a write comes from the vm, the backup blocks that until it has backed up the current state of that block on disk (if that block was not already backed up)
 
May 24, 2022
7
0
1
Hello! I mean during the backup the VM is totally unreachable! No pingable! We can't do nothing! But at the moment of the backup is finish or stopped, everything go back normally.
 

dcsapak

Proxmox Staff Member
Staff member
Feb 1, 2016
7,874
952
163
33
Vienna
can you post the task log of such a backup?
 
May 24, 2022
7
0
1
Task viewer: VM/CT 103 - Backup


OutputStatus

Stop
INFO: starting new backup job: vzdump 103 --node pve1 --notes-template '{{guestname}}' --mode snapshot --storage PMBackup2 --remove 0 --compress zstd
INFO: Starting Backup of VM 103 (qemu)
INFO: Backup started at 2022-05-29 17:42:20
INFO: status = running
INFO: VM Name: PleskTython
INFO: include disk 'scsi0' 'DCSAN:103/vm-103-disk-0.qcow2' 150G
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating vzdump archive '/mnt/pve/PMBackup2/dump/vzdump-qemu-103-2022_05_29-17_42_20.vma.zst'
INFO: issuing guest-agent 'fs-freeze' command
INFO: issuing guest-agent 'fs-thaw' command
INFO: started backup task 'd1cc8b26-467a-4750-a492-655c5398338c'
INFO: resuming VM again
INFO: 0% (348.0 MiB of 150.0 GiB) in 3s, read: 116.0 MiB/s, write: 108.4 MiB/s
INFO: 1% (1.6 GiB of 150.0 GiB) in 14s, read: 117.3 MiB/s, write: 44.7 MiB/s
INFO: 2% (3.0 GiB of 150.0 GiB) in 27s, read: 112.7 MiB/s, write: 29.4 MiB/s
INFO: 3% (4.6 GiB of 150.0 GiB) in 41s, read: 112.4 MiB/s, write: 128.3 KiB/s
INFO: 4% (6.1 GiB of 150.0 GiB) in 55s, read: 112.3 MiB/s, write: 72.2 MiB/s
INFO: 5% (7.5 GiB of 150.0 GiB) in 1m 8s, read: 111.6 MiB/s, write: 110.7 MiB/s
ERROR: interrupted by signal
INFO: aborting backup job
INFO: resuming VM again
ERROR: Backup of VM 103 failed - interrupted by signal
 

dcsapak

Proxmox Staff Member
Staff member
Feb 1, 2016
7,874
952
163
33
Vienna
i mean it probably does not have to with your problem, but you don't backup to a pbs but a regular vzdump backup? (since the thread was about pbs)

as i said, during backup, the writes from inside the vm to blocks that were not backed up yet, must be paused (until they are backed up)
maybe the backup does saturate your network and the vm cannot access the disks anymore ?

can you post the vm config and storage config too ?
 
May 24, 2022
7
0
1
The problem is the same with PBS or VZDUMP!
We are trying now the KVM with backup and no problem! We have try too with ESXI Vmware no problem!

The problem is ProxMox
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get your own in 60 seconds.

Buy now!