Slow Backup reading source

[scasrl]

Hello,
I have a pbs server for backup vm of a pve cluster, but, when i start back big vm disk from ceph storage the maximum read speed is about 40-50 MB/s for the first backup.
If i start a new stopped backup i have ever the same speed...

INFO: 0% (116.0 MiB of 332.0 GiB) in 3s, read: 38.7 MiB/s, write: 38.7 MiB/s
INFO: 1% (3.3 GiB of 332.0 GiB) in 1m 5s, read: 53.4 MiB/s, write: 53.4 MiB/s
INFO: 2% (6.6 GiB of 332.0 GiB) in 2m 22s, read: 43.9 MiB/s, write: 43.9 MiB/s

P.S. the ceph storage is connected with bond of 2x10GB

Uploaded 448 chunks in 5 seconds.
Time per request: 11222 microseconds.
TLS speed: 373.74 MB/s
SHA256 speed: 318.07 MB/s
Compression speed: 489.56 MB/s
Decompress speed: 809.77 MB/s
AES256/GCM speed: 1952.62 MB/s
Verify speed: 258.55 MB/s
┌───────────────────────────────────┬────────────────────┐
│ Name │ Value │
╞═══════════════════════════════════╪════════════════════╡
│ TLS (maximal backup upload speed) │ 373.74 MB/s (30%) │
├───────────────────────────────────┼────────────────────┤
│ SHA256 checksum computation speed │ 318.07 MB/s (16%) │
├───────────────────────────────────┼────────────────────┤
│ ZStd level 1 compression speed │ 489.56 MB/s (65%) │
├───────────────────────────────────┼────────────────────┤
│ ZStd level 1 decompression speed │ 809.77 MB/s (68%) │
├───────────────────────────────────┼────────────────────┤
│ Chunk verification speed │ 258.55 MB/s (34%) │
├───────────────────────────────────┼────────────────────┤
│ AES256 GCM encryption speed │ 1952.62 MB/s (54%) │
└───────────────────────────────────┴────────────────────┘

 

[tom]

Looks like your VM storage is just slow.?

=> Benchmark your Ceph

 

[scasrl]

This is the bonnie++ benchmark inside a debian container, why I have only 50MiB/s in read with Proxmox Backup Server?

Version  1.98       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Name:Size etc        /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
TEST            16G            525m  63  239m  48            774m  50  2261  90
Latency                       32649us     214ms             10424us   14354us

 

[Digitaldaz]

On my small cluster I also see a huge difference between PBS and NFS backup storage with Ceph.

This is a 25GB VM

To PBS - Over two and a half hours:


Same VM to NFS using a regular backup, no PBS - Just over three minutes:

 

[scasrl]

I tried to start two backups simultaneously, of two different VMs, from the same ceph storage and the speed is 50MiB/s for each but 100MiB/s for the total reading of the two processes
It seems to have limits on reading data for each backup ... Is this possible?

 

[scasrl]

Can anyone give me an explanation? Or where to check?

 

[mmidgett]

I was hitting this 50MB mark myself and I was told that it was an AES problem with my processor on the storage server. I my case the PBS server had old L5520 Xeons that didn't support AES so I ebayed some X5670's that will be here next week. I don't know if this is your problem or not.

https://forum.proxmox.com/threads/increasing-backup-speed.86436/

 

[scasrl]

mmmm... the pbs is in a container so with the host cpu, the client is on a recent hpe server with definitely has aes activated ... I don't think that's the problem

 

[scasrl]

This is the log of a backup of a vm that has been stopped and restarted and, as you can see, the writing is reduced but the reading is always around 50MiB/s is it not possible to increase the reading speed?

INFO: resuming VM again after 13 seconds
INFO: scsi0: dirty-bitmap status: created new
INFO: scsi1: dirty-bitmap status: created new
INFO: 0% (156.0 MiB of 432.0 GiB) in 3s, read: 52.0 MiB/s, write: 28.0 MiB/s
INFO: 1% (4.3 GiB of 432.0 GiB) in 1m 8s, read: 65.8 MiB/s, write: 7.5 MiB/s
INFO: 2% (8.7 GiB of 432.0 GiB) in 2m 41s, read: 47.6 MiB/s, write: 7.0 MiB/s
INFO: 3% (13.0 GiB of 432.0 GiB) in 4m 8s, read: 50.7 MiB/s, write: 4.3 MiB/s
INFO: 4% (17.3 GiB of 432.0 GiB) in 5m 45s, read: 46.0 MiB/s, write: 2.8 MiB/s
INFO: 5% (21.6 GiB of 432.0 GiB) in 7m 19s, read: 47.0 MiB/s, write: 2.0 MiB/s
INFO: 6% (26.0 GiB of 432.0 GiB) in 8m 51s, read: 48.3 MiB/s, write: 2.3 MiB/s
INFO: 7% (30.3 GiB of 432.0 GiB) in 10m 22s, read: 48.5 MiB/s, write: 1.3 MiB/s
INFO: 8% (34.6 GiB of 432.0 GiB) in 11m 55s, read: 47.7 MiB/s, write: 2.6 MiB/s
INFO: 9% (38.9 GiB of 432.0 GiB) in 13m 18s, read: 52.8 MiB/s, write: 740.2 KiB/s
INFO: 10% (43.2 GiB of 432.0 GiB) in 14m 45s, read: 51.0 MiB/s, write: 1.5 MiB/s
INFO: 11% (47.6 GiB of 432.0 GiB) in 16m 12s, read: 51.3 MiB/s, write: 847.4 KiB/s
INFO: 12% (51.9 GiB of 432.0 GiB) in 17m 41s, read: 49.4 MiB/s, write: 460.2 KiB/s
INFO: 13% (56.2 GiB of 432.0 GiB) in 19m 14s, read: 47.4 MiB/s, write: 1.3 MiB/s
INFO: 14% (60.5 GiB of 432.0 GiB) in 20m 45s, read: 48.7 MiB/s, write: 225.1 KiB/s