Slow Backup reading source

Apr 23, 2020
25
3
8
Hello,
I have a pbs server for backup vm of a pve cluster, but, when i start back big vm disk from ceph storage the maximum read speed is about 40-50 MB/s for the first backup.
If i start a new stopped backup i have ever the same speed...

INFO: 0% (116.0 MiB of 332.0 GiB) in 3s, read: 38.7 MiB/s, write: 38.7 MiB/s
INFO: 1% (3.3 GiB of 332.0 GiB) in 1m 5s, read: 53.4 MiB/s, write: 53.4 MiB/s
INFO: 2% (6.6 GiB of 332.0 GiB) in 2m 22s, read: 43.9 MiB/s, write: 43.9 MiB/s

P.S. the ceph storage is connected with bond of 2x10GB

Uploaded 448 chunks in 5 seconds.
Time per request: 11222 microseconds.
TLS speed: 373.74 MB/s
SHA256 speed: 318.07 MB/s
Compression speed: 489.56 MB/s
Decompress speed: 809.77 MB/s
AES256/GCM speed: 1952.62 MB/s
Verify speed: 258.55 MB/s
┌───────────────────────────────────┬────────────────────┐
│ Name │ Value │
╞═══════════════════════════════════╪════════════════════╡
│ TLS (maximal backup upload speed) │ 373.74 MB/s (30%) │
├───────────────────────────────────┼────────────────────┤
│ SHA256 checksum computation speed │ 318.07 MB/s (16%) │
├───────────────────────────────────┼────────────────────┤
│ ZStd level 1 compression speed │ 489.56 MB/s (65%) │
├───────────────────────────────────┼────────────────────┤
│ ZStd level 1 decompression speed │ 809.77 MB/s (68%) │
├───────────────────────────────────┼────────────────────┤
│ Chunk verification speed │ 258.55 MB/s (34%) │
├───────────────────────────────────┼────────────────────┤
│ AES256 GCM encryption speed │ 1952.62 MB/s (54%) │
└───────────────────────────────────┴────────────────────┘
 
Last edited:
Looks like your VM storage is just slow.?

=> Benchmark your Ceph
 
This is the bonnie++ benchmark inside a debian container, why I have only 50MiB/s in read with Proxmox Backup Server?

Code:
Version  1.98       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Name:Size etc        /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
TEST            16G            525m  63  239m  48            774m  50  2261  90
Latency                       32649us     214ms             10424us   14354us
 
Last edited:
On my small cluster I also see a huge difference between PBS and NFS backup storage with Ceph.

This is a 25GB VM

To PBS - Over two and a half hours:
1616236229453.png

Same VM to NFS using a regular backup, no PBS - Just over three minutes:
1616236408939.png
 

Attachments

  • 1616236369731.png
    1616236369731.png
    5 KB · Views: 30
I tried to start two backups simultaneously, of two different VMs, from the same ceph storage and the speed is 50MiB/s for each but 100MiB/s for the total reading of the two processes
It seems to have limits on reading data for each backup ... Is this possible?
 
Last edited:
I was hitting this 50MB mark myself and I was told that it was an AES problem with my processor on the storage server. I my case the PBS server had old L5520 Xeons that didn't support AES so I ebayed some X5670's that will be here next week. I don't know if this is your problem or not.

https://forum.proxmox.com/threads/increasing-backup-speed.86436/
 
Last edited:
mmmm... the pbs is in a container so with the host cpu, the client is on a recent hpe server with definitely has aes activated ... I don't think that's the problem
 
This is the log of a backup of a vm that has been stopped and restarted and, as you can see, the writing is reduced but the reading is always around 50MiB/s is it not possible to increase the reading speed?

INFO: resuming VM again after 13 seconds
INFO: scsi0: dirty-bitmap status: created new
INFO: scsi1: dirty-bitmap status: created new
INFO: 0% (156.0 MiB of 432.0 GiB) in 3s, read: 52.0 MiB/s, write: 28.0 MiB/s
INFO: 1% (4.3 GiB of 432.0 GiB) in 1m 8s, read: 65.8 MiB/s, write: 7.5 MiB/s
INFO: 2% (8.7 GiB of 432.0 GiB) in 2m 41s, read: 47.6 MiB/s, write: 7.0 MiB/s
INFO: 3% (13.0 GiB of 432.0 GiB) in 4m 8s, read: 50.7 MiB/s, write: 4.3 MiB/s
INFO: 4% (17.3 GiB of 432.0 GiB) in 5m 45s, read: 46.0 MiB/s, write: 2.8 MiB/s
INFO: 5% (21.6 GiB of 432.0 GiB) in 7m 19s, read: 47.0 MiB/s, write: 2.0 MiB/s
INFO: 6% (26.0 GiB of 432.0 GiB) in 8m 51s, read: 48.3 MiB/s, write: 2.3 MiB/s
INFO: 7% (30.3 GiB of 432.0 GiB) in 10m 22s, read: 48.5 MiB/s, write: 1.3 MiB/s
INFO: 8% (34.6 GiB of 432.0 GiB) in 11m 55s, read: 47.7 MiB/s, write: 2.6 MiB/s
INFO: 9% (38.9 GiB of 432.0 GiB) in 13m 18s, read: 52.8 MiB/s, write: 740.2 KiB/s
INFO: 10% (43.2 GiB of 432.0 GiB) in 14m 45s, read: 51.0 MiB/s, write: 1.5 MiB/s
INFO: 11% (47.6 GiB of 432.0 GiB) in 16m 12s, read: 51.3 MiB/s, write: 847.4 KiB/s
INFO: 12% (51.9 GiB of 432.0 GiB) in 17m 41s, read: 49.4 MiB/s, write: 460.2 KiB/s
INFO: 13% (56.2 GiB of 432.0 GiB) in 19m 14s, read: 47.4 MiB/s, write: 1.3 MiB/s
INFO: 14% (60.5 GiB of 432.0 GiB) in 20m 45s, read: 48.7 MiB/s, write: 225.1 KiB/s
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!