If I upgrade to 10/20 Gbit, would the speed increase?

fireon

Famous Member
Oct 25, 2010
3,509
264
103
38
Austria/Graz
iteas.at
Hello all,

i've done some tests with the backup. Always the whole Cluster. Nobody works on the cluster during this time. What i have tested:

  • Fullbackup with normal vzdump over Gigabit form ZFS Raid10 to an NFS4.2 Share on the Backupserver with 8HDD's in RaidZ10 with compression ZST. Size 102.68GB in 00:40:07
  • The First Fullbackup with new PBS. Size 334.63GB in 00:40:07

I noticed that the network with PBS was never exhausted, always only 20 to 50MB / s, fluctuating. For this reason also my question. If I upgrade to 10/20 Gbit, would the speed increase?

If i do an normal ZFS SEND copy it about 120MB/s. How is the whole to be understood?

I've started one really big VM with 7TB. I've aborted the process after some hours. It was so slow. At least it looked like that. Here es the Log:
Code:
118: 2020-09-06 01:14:40 INFO: Starting Backup of VM 118 (qemu)
118: 2020-09-06 01:14:40 INFO: status = running
118: 2020-09-06 01:14:40 INFO: VM Name: data.tux.lan
118: 2020-09-06 01:14:40 INFO: include disk 'scsi0' 'SSD-vmdata:vm-118-disk-1' 30G
118: 2020-09-06 01:14:40 INFO: include disk 'scsi1' 'SSD-vmdata:vm-118-disk-2' 8G
118: 2020-09-06 01:14:40 INFO: include disk 'scsi2' 'HDD-vmdata:vm-118-disk-3' 7000G
118: 2020-09-06 01:14:40 INFO: include disk 'efidisk0' 'SSD-vmdata:vm-118-disk-0' 128K
118: 2020-09-06 01:14:40 INFO: backup mode: snapshot
118: 2020-09-06 01:14:40 INFO: ionice priority: 7
118: 2020-09-06 01:14:40 INFO: creating Proxmox Backup Server archive 'vm/118/2020-09-05T23:14:40Z'
118: 2020-09-06 01:14:40 INFO: issuing guest-agent 'fs-freeze' command
118: 2020-09-06 01:14:41 INFO: issuing guest-agent 'fs-thaw' command
118: 2020-09-06 01:14:41 INFO: started backup task '8703edfe-c042-41a3-9da8-f56b4bc3b040'
118: 2020-09-06 01:14:41 INFO: resuming VM again
118: 2020-09-06 01:14:41 INFO: efidisk0: dirty-bitmap status: existing bitmap was invalid and has been cleared
118: 2020-09-06 01:14:41 INFO: scsi0: dirty-bitmap status: existing bitmap was invalid and has been cleared
118: 2020-09-06 01:14:41 INFO: scsi1: dirty-bitmap status: existing bitmap was invalid and has been cleared
118: 2020-09-06 01:14:41 INFO: scsi2: dirty-bitmap status: created new
118: 2020-09-06 01:14:44 INFO:   0% (1.2 GiB of 6.9 TiB) in  3s, read: 409.4 MiB/s, write: 82.7 MiB/s
118: 2020-09-06 01:47:26 INFO:   1% (70.4 GiB of 6.9 TiB) in 32m 45s, read: 36.1 MiB/s, write: 34.0 MiB/s
118: 2020-09-06 02:51:59 INFO:   2% (140.8 GiB of 6.9 TiB) in  1h 37m 18s, read: 18.6 MiB/s, write: 17.4 MiB/s
118: 2020-09-06 03:25:00 INFO:   3% (211.2 GiB of 6.9 TiB) in  2h 10m 19s, read: 36.4 MiB/s, write: 34.2 MiB/s
118: 2020-09-06 04:05:06 INFO:   4% (281.5 GiB of 6.9 TiB) in  2h 50m 25s, read: 29.9 MiB/s, write: 28.1 MiB/s
118: 2020-09-06 04:39:58 INFO:   5% (351.9 GiB of 6.9 TiB) in  3h 25m 17s, read: 34.5 MiB/s, write: 32.4 MiB/s
118: 2020-09-06 05:16:08 INFO:   6% (422.3 GiB of 6.9 TiB) in  4h  1m 27s, read: 33.2 MiB/s, write: 31.2 MiB/s
118: 2020-09-06 05:57:23 INFO:   7% (492.7 GiB of 6.9 TiB) in  4h 42m 42s, read: 29.1 MiB/s, write: 27.4 MiB/s
118: 2020-09-06 06:37:33 INFO:   8% (563.1 GiB of 6.9 TiB) in  5h 22m 52s, read: 29.9 MiB/s, write: 27.6 MiB/s
118: 2020-09-06 07:23:03 INFO:   9% (633.4 GiB of 6.9 TiB) in  6h  8m 22s, read: 26.4 MiB/s, write: 24.8 MiB/s
118: 2020-09-06 08:08:47 INFO:  10% (703.8 GiB of 6.9 TiB) in  6h 54m  6s, read: 26.3 MiB/s, write: 24.6 MiB/s
118: 2020-09-06 08:56:30 INFO:  11% (774.2 GiB of 6.9 TiB) in  7h 41m 49s, read: 25.2 MiB/s, write: 23.7 MiB/s
118: 2020-09-06 09:43:35 INFO:  12% (844.6 GiB of 6.9 TiB) in  8h 28m 54s, read: 25.5 MiB/s, write: 24.0 MiB/s
118: 2020-09-06 10:29:04 INFO:  13% (914.9 GiB of 6.9 TiB) in  9h 14m 23s, read: 26.4 MiB/s, write: 24.8 MiB/s
118: 2020-09-06 11:15:14 INFO:  14% (985.4 GiB of 6.9 TiB) in 10h  0m 33s, read: 26.0 MiB/s, write: 24.5 MiB/s
118: 2020-09-06 12:01:01 INFO:  15% (1.0 TiB of 6.9 TiB) in 10h 46m 20s, read: 26.2 MiB/s, write: 24.8 MiB/s
118: 2020-09-06 12:47:04 INFO:  16% (1.1 TiB of 6.9 TiB) in 11h 32m 23s, read: 26.1 MiB/s, write: 24.7 MiB/s
118: 2020-09-06 13:32:11 INFO:  17% (1.2 TiB of 6.9 TiB) in 12h 17m 30s, read: 26.6 MiB/s, write: 25.2 MiB/s
118: 2020-09-06 13:42:59 ERROR: interrupted by signal
118: 2020-09-06 13:42:59 INFO: aborting backup job
118: 2020-09-06 13:42:59 ERROR: Backup of VM 118 failed - interrupted by signal
Backup Speedtest:
Code:
proxmox-backup-client benchmark
Password for "xxxxxxxxx@pbs": ******************
Uploaded 147 chunks in 5 seconds.
Time per request: 36374 microseconds.
TLS speed: 115.31 MB/s
SHA256 speed: 1382.31 MB/s
Compression speed: 1525.76 MB/s
Decompress speed: 7045.81 MB/s
AES256/GCM speed: 2523.92 MB/s
┌───────────────────────────────────┬────────────────────┐
│ Name                              │ Value              │
╞═══════════════════════════════════╪════════════════════╡
│ TLS (maximal backup upload speed) │ 115.31 MB/s (20%)  │
├───────────────────────────────────┼────────────────────┤
│ SHA256 checksum computation speed │ 1382.31 MB/s (65%) │
├───────────────────────────────────┼────────────────────┤
│ ZStd level 1 compression speed    │ 1525.76 MB/s (71%) │
├───────────────────────────────────┼────────────────────┤
│ ZStd level 1 decompression speed  │ 7045.81 MB/s (87%) │
├───────────────────────────────────┼────────────────────┤
│ AES256 GCM encryption speed       │ 2523.92 MB/s (66%) │
└───────────────────────────────────┴────────────────────┘
proxmox-backup-client benchmark  7,42s user 0,44s system 30% cpu 25,781 total
 

fabian

Proxmox Staff Member
Staff member
Jan 7, 2016
4,460
696
133
we found some bottlenecks related to TLS/HTTP2 speed in some environments, it would be great if you could test upgraded packages once they are released!
 

fireon

Famous Member
Oct 25, 2010
3,509
264
103
38
Austria/Graz
iteas.at
we found some bottlenecks related to TLS/HTTP2 speed in some environments, it would be great if you could test upgraded packages once they are released!
Yeah... absolutly. I wait to backup my 7TB VM on external disk. Tell me if i should upgrade and test.
 

fabian

Proxmox Staff Member
Staff member
Jan 7, 2016
4,460
696
133
proxmox-backup-server and -client >= 0.8.15-1 contain the changes (bumped, but not yet on pbstest/pvetest), and libproxmox-backup-qemu0 >= 0.6.5 should contain it as well (not yet updated and bumped). for VMs, you need to stop and start the VM to load the updated version of the backup library.
 

fabian

Proxmox Staff Member
Staff member
Jan 7, 2016
4,460
696
133
now available on pbstest / pve-no-subscription. there is one more improvement to the TLS speed that is not yet part of the packaged version, but maybe you can try running the benchmark now and then again once that hits the repos?
 

fireon

Famous Member
Oct 25, 2010
3,509
264
103
38
Austria/Graz
iteas.at
Doesn't seem to work anymore.

Code:
export PBS_REPOSITORY="backupuser@pbs@tuxi:backupshare"
proxmox-backup-client benchmark
Error: unable to run benchmark without --benchmark flags
 

fabian

Proxmox Staff Member
Staff member
Jan 7, 2016
4,460
696
133
you need to update both client and server (the benchmark now marks benchmark runs with a special flag, and the benchmark backup ID is reserved for backups with that flag so that no accidental name collision can happen)
 

fireon

Famous Member
Oct 25, 2010
3,509
264
103
38
Austria/Graz
iteas.at
Here the new one after the Upgrade:
Code:
Uploaded 146 chunks in 5 seconds.
Time per request: 35964 microseconds.
TLS speed: 116.62 MB/s
SHA256 speed: 1373.66 MB/s
Compression speed: 1496.64 MB/s
Decompress speed: 6439.88 MB/s
AES256/GCM speed: 2730.96 MB/s
┌───────────────────────────────────┬────────────────────┐
│ Name                              │ Value              │
╞═══════════════════════════════════╪════════════════════╡
│ TLS (maximal backup upload speed) │ 116.62 MB/s (17%)  │
├───────────────────────────────────┼────────────────────┤
│ SHA256 checksum computation speed │ 1373.66 MB/s (65%) │
├───────────────────────────────────┼────────────────────┤
│ ZStd level 1 compression speed    │ 1496.64 MB/s (69%) │
├───────────────────────────────────┼────────────────────┤
│ ZStd level 1 decompression speed  │ 6439.88 MB/s (80%) │
├───────────────────────────────────┼────────────────────┤
│ AES256 GCM encryption speed       │ 2730.96 MB/s (72%) │
└───────────────────────────────────┴────────────────────┘
proxmox-backup-client benchmark  7,21s user 0,33s system 34% cpu 21,786 total
 

fabian

Proxmox Staff Member
Staff member
Jan 7, 2016
4,460
696
133
how fast is HDD-vmdata when you benchmark inside the VM? how fast from the host?
 

fireon

Famous Member
Oct 25, 2010
3,509
264
103
38
Austria/Graz
iteas.at
How can i do this? Pveperf did not work on this node. No error. Output only to "HD SIZE". After that nothing. After 1h noting. The only way is to kill the command. On the other both nodes pveperf do the job.

So how can i do this on the host, and on the VM. I remember... if scrub on ZFS is running, there about 800MB/s.
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE and Proxmox Mail Gateway. We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get your own in 60 seconds.

Buy now!