Backup and Restore on current machine - searching bottle neck

[grefabu]

I want to setup an PVE/PBS Server on the same machine.
We have an HP DL380 Gen9 with

E5-2640 v3 @ 2.60GHz
256 GB RAM

2x 1TB SAMSUNG NVME PM983
12x 8 TB HP SAS HDDs

The goal is an backup server which could be even a desaster restore virtual server.

Unfortunaly I never got good restore rates. even between 40 and 70 MB/s.
The last test is, I define the nvmes one as vmstore and one as backup store (over pbs), both as mount with ext4 as file system. So there is no harm from zfs,...

The restore rate is only 70 MB/s.

Our HW seller, which although runs proxmox tells it is an case of comprimysation and the single core performance:

This has nothing to do with the server or hardware defects - it is due to the gzip LZO etc.... compression method. By default, only one core is used per task.

We have the same backup speeds in our cluster

But I guess the zstd should be multicore as default?
But when I run an restore it seem that one core is nearly utilized, even it 6 task?

 

[grefabu]

I made a new test with an other machine:

Xeon(R) CPU E3-1246 v3 @ 3.50GHz
16 GB RAM

1x Seagate Ironwolf PRO ST4000 4TB SATA
2x 1TB SAMSUNG NVME PM983
1x Seagate Ironwolf PRO ST6000 6TB SATA


The 4TB ST400 is for OS
The 6TB ST6000 is for the PBS Backupstore
One NVME is for PVE VMStore
One NVME is for a second PBS Backupstore

I Synch from our productiv PBS to the &TB Backupstore and made an Restore with the one nvme as goal:

Formatting '/mnt/pve/VMSTORENVME/images/121/vm-121-disk-0.raw', fmt=raw size=536870912000
new volume ID is 'VMSTORENVME:121/vm-121-disk-0.raw'
restore proxmox backup image: /usr/bin/pbs-restore --repository root@pam@localhost:BACKUPSATA6TB vm/121/2021-09-07T19:00:02Z drive-virtio0.img.fidx /mnt/pve/VMSTORENVME/images/121/vm-121-disk-0.raw --verbose --format raw --skip-zero
connecting to repository 'root@pam@localhost:BACKUPSATA6TB'
open block backend for target '/mnt/pve/VMSTORENVME/images/121/vm-121-disk-0.raw'
starting to restore snapshot 'vm/121/2021-09-07T19:00:02Z'
download and verify backup index
progress 1% (read 5368709120 bytes, zeroes = 2% (121634816 bytes), duration 39 sec)

progress 99% (read 531502202880 bytes, zeroes = 0% (2898264064 bytes), duration 4777 sec)
progress 100% (read 536870912000 bytes, zeroes = 0% (2898264064 bytes), duration 4821 sec)
restore image complete (bytes=536870912000, duration=4821.47s, speed=106.19MB/s)
rescan volumes...
TASK OK

The rate is pretty better than the rate on the DL380 with the E5-2640 v3 @ 2.60GHz
When I store from the nvme to nvme the rate is about 180 MB/s. Not as good as the raw NVME rate with 1.2 GB/s (messure with dd) but 'not bad'

But where is the real bottle neck? The CPU?

I've here some AMD Ryzen PCs. Maybee the next Test is with one of these.

 

[grefabu]

Hi,

I'm still searching.
On an AMD Ryzen PC the rate is even better, sorry no result yet,..

But what I additional see on the original Server:

When I start multipe restore jobs parallel the iorate raise to 150-200 MB/s, but the rate of an single job is still on 40-50 MB/s:


On this machine I made an backup benchmark:
root@ph-pbs:~# proxmox-backup-client benchmark
SHA256 speed: 398.95 MB/s
Compression speed: 424.26 MB/s
Decompress speed: 1139.92 MB/s
AES256/GCM speed: 1990.74 MB/s
Verify speed: 315.00 MB/s
┌───────────────────────────────────┬────────────────────┐
│ Name │ Value │
╞═══════════════════════════════════╪════════════════════╡
│ TLS (maximal backup upload speed) │ not tested │
├───────────────────────────────────┼────────────────────┤
│ SHA256 checksum computation speed │ 398.95 MB/s (20%) │
├───────────────────────────────────┼────────────────────┤
│ ZStd level 1 compression speed │ 424.26 MB/s (56%) │
├───────────────────────────────────┼────────────────────┤
│ ZStd level 1 decompression speed │ 1139.92 MB/s (95%) │
├───────────────────────────────────┼────────────────────┤
│ Chunk verification speed │ 315.00 MB/s (42%) │
├───────────────────────────────────┼────────────────────┤
│ AES256 GCM encryption speed │ 1990.74 MB/s (55%) │
└───────────────────────────────────┴────────────────────┘

So the bottle neck isn't the HDDs but what?

 

[grefabu]

I made a new test with the HP DL380 an an other CPU:

Intel(R) Xeon(R) CPU E5-2643 v3 @ 3.40GHz

Formatting '/ZFSPOOL/VMSTORE001/images/121/vm-121-disk-0.raw', fmt=raw size=536870912000
new volume ID is 'VMSTORE001:121/vm-121-disk-0.raw'
restore proxmox backup image: /usr/bin/pbs-restore --repository root@pam@ph-pbs.peiker-holding.de:BACKUPSTORE001 vm/121/2021-09-28T19:00:03Z drive-virtio0.img.fidx /ZFSPOOL/VMSTORE001/images/121/vm-121-disk-0.raw --verbose --format raw --skip-zero
connecting to repository 'root@pam@ph-pbs.peiker-holding.de:BACKUPSTORE001'
open block backend for target '/ZFSPOOL/VMSTORE001/images/121/vm-121-disk-0.raw'
starting to restore snapshot 'vm/121/2021-09-28T19:00:03Z'
download and verify backup index
progress 1% (read 5368709120 bytes, zeroes = 1% (104857600 bytes), duration 92 sec)
progress 2% (read 10737418240 bytes, zeroes = 1% (138412032 bytes), duration 189 sec)
progress 3% (read 16106127360 bytes, zeroes = 0% (138412032 bytes), duration 291 sec)
.
.
.
progress 99% (read 531502202880 bytes, zeroes = 0% (2243952640 bytes), duration 11195 sec)
progress 100% (read 536870912000 bytes, zeroes = 0% (2243952640 bytes), duration 11289 sec)
restore image complete (bytes=536870912000, duration=11289.28s, speed=45.35MB/s)
rescan volumes...
TASK OK

Although the CPU single threat is oblivious more than the E5-2640 v3 @ 2.60GHz the restore rate is still in an low level.
Thoug when the CPU isn't the resaon, and the file system too (parallel jobs run the IO still over the limit), what could it bee?

The RAM frequenzy?