slowness

m!ka · Mar 29, 2021

Hello,

I just set up a 1st PBS and I have very poor performance:

Code:

Uploaded 99 chunks in 5 seconds.
Time per request: 51253 microseconds.
TLS speed: 81.83 MB/s
SHA256 speed: 188.74 MB/s
Compression speed: 317.88 MB/s
Decompress speed: 582.48 MB/s
AES256/GCM speed: 112.44 MB/s
Verify speed: 155.37 MB/s
┌───────────────────────────────────┬───────────────────┐
│ Name                              │ Value             │
╞═══════════════════════════════════╪═══════════════════╡
│ TLS (maximal backup upload speed) │ 81.83 MB/s (7%)   │
├───────────────────────────────────┼───────────────────┤
│ SHA256 checksum computation speed │ 188.74 MB/s (9%)  │
├───────────────────────────────────┼───────────────────┤
│ ZStd level 1 compression speed    │ 317.88 MB/s (42%) │
├───────────────────────────────────┼───────────────────┤
│ ZStd level 1 decompression speed  │ 582.48 MB/s (49%) │
├───────────────────────────────────┼───────────────────┤
│ Chunk verification speed          │ 155.37 MB/s (20%) │
├───────────────────────────────────┼───────────────────┤
│ AES256 GCM encryption speed       │ 112.44 MB/s (3%)  │
└───────────────────────────────────┴───────────────────┘

Can you help me?

Hardware:

Code:

  pool: backup
state: ONLINE
config:

    NAME        STATE     READ WRITE CKSUM
    backup      ONLINE       0     0     0
      raidz1-0  ONLINE       0     0     0
        sdc     ONLINE       0     0     0
        sdd     ONLINE       0     0     0
        sde     ONLINE       0     0     0
        sdf     ONLINE       0     0     0
        sdg     ONLINE       0     0     0
    logs  
      sdh       ONLINE       0     0     0

sdc-sdg : 5x SATA Western Digital WD Gold 6 To (WD6003FRYZ-0)
sdh: Crucial SSD M4 CT064M4SSD2 64Go
sda-sdb: Proxmox system (2x Sandisk 128Go SSD mirror-0)

CPU: 2x Intel(R) Xeon(R) CPU E5620 @ 2.40GHz (4 cores, 8 threads)
Memory: 72472Mo DDR3
Motherboard: SuperMicro X8DTN+
Adapter: LSI 6Gbps SAS HBA LSI 9200-8i = (9211-8I) IT Mode
Backplane: BPN-SAS2-826EL1
LAN: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection

tom · Mar 29, 2021

m!ka said:
I just set up a 1st PBS and I have very poor performance:

You CPU is really outdated (released 11 years ago, https://ark.intel.com/content/www/d...0-12m-cache-2-40-ghz-5-86-gt-s-intel-qpi.html)

I really recommend to use modern CPU and fast storage only (SSDs, NVMe) for Proxmox Backup Server.

spirit · Mar 29, 2021

aes,tls will be slow, as your cpu don't have hardware support for it.

also, raidz-1 can be slow for write, as you'll have almost the speed of 1 disk, because of parity. (better to do a raid10 with bigger disks, or use a good ssd for zfs log device.
(I'm not sure if your Crucial SSD M4 CT064M4SSD2 64Go is for log device, but you should really use a good datacenter ssd, because this crucial is pretty low for sync write)

m!ka · Mar 29, 2021

tom said:
You CPU is really outdated (released 11 years ago, https://ark.intel.com/content/www/d...0-12m-cache-2-40-ghz-5-86-gt-s-intel-qpi.html)

I really recommend to use modern CPU and fast storage only (SSDs, NVMe) for Proxmox Backup Server.

Exactly, 2 CPUs are old, but they are only used at 11% while the transfer speed tops out at 500-600Mbit/s.

tom · Mar 29, 2021

m!ka said:
500-600Mbit/s.

Your PBS benchmarks do not show these numbers.

The "slowness" is expected for this too old hardware.

m!ka · Mar 29, 2021

tom said:
Your PBS benchmarks do not show these numbers.

The "slowness" is expected for this too old hardware.

TLS (maximal backup upload speed) │ 81.83 MB/s

81.83 MB/s * 8 = 654Mbit/s

Confirmed by "nload":

Capture d’écran 2021-03-29 à 16.52.42.png

tom · Mar 29, 2021

ah sorry, I misread your benchmark data.

Compare your results to current CPU models:

https://forum.proxmox.com/threads/how-fast-is-your-backup-datastore-benchmark-tool.72750/

m!ka · Mar 29, 2021

spirit said:
aes,tls will be slow, as your cpu don't have hardware support for it.

also, raidz-1 can be slow for write, as you'll have almost the speed of 1 disk, because of parity. (better to do a raid10 with bigger disks, or use a good ssd for zfs log device.
(I'm not sure if your Crucial SSD M4 CT064M4SSD2 64Go is for log device, but you should really use a good datacenter ssd, because this crucial is pretty low for sync write

Hello spirit,

I will reconfigure my disks in Raid10 and do some tests.

For the CPU .. are you sure AES is not supported? Intel says this:

Intel® AES New Instructions Yes

For the SSD log, I did not see any improvement with it, and when I monitor with iostat, I do not see any activity.

m!ka · Mar 29, 2021

@spirit

indeed, the performance of the Crucial is not very good in writing:

Sustained Sequential Write up to (128k transfer) 95MB/s
Sustained Sequential Read up to (128k transfer) 500MB/s

I read that we recommend a small capacity SSD drive, and I took the 1st small size drive that I found in my stock without paying attention to its performance.

would a Samsung SM883 be better?

Seq. Read 540 MB/s
Seq. Write 480 MB/s

or is the NVME mandatory?

Afterwards, I may be wrong, but via 'iostat' I don't see my Crucial really working ... I wonder if it is useful ...

m!ka · Apr 2, 2021

Hello,

I replaced the SSD for ZFS log by "Samsung SM883 240GB 2.5in Data Center SATA 6Gbps"

Code:

Seq. Read    540 MB/s    Seq. Write    480 MB/s
Ran. Read    97K IOPS    Ran. Write    22K IOPS

Not really a big improvement...

Code:

Uploaded 110 chunks in 5 seconds.
Time per request: 46514 microseconds.
TLS speed: 90.17 MB/s
SHA256 speed: 192.15 MB/s
Compression speed: 332.31 MB/s
Decompress speed: 528.12 MB/s
AES256/GCM speed: 101.22 MB/s
Verify speed: 155.31 MB/s
┌───────────────────────────────────┬───────────────────┐
│ Name                              │ Value             │
╞═══════════════════════════════════╪═══════════════════╡
│ TLS (maximal backup upload speed) │ 90.17 MB/s (7%)   │
├───────────────────────────────────┼───────────────────┤
│ SHA256 checksum computation speed │ 192.15 MB/s (10%) │
├───────────────────────────────────┼───────────────────┤
│ ZStd level 1 compression speed    │ 332.31 MB/s (44%) │
├───────────────────────────────────┼───────────────────┤
│ ZStd level 1 decompression speed  │ 528.12 MB/s (44%) │
├───────────────────────────────────┼───────────────────┤
│ Chunk verification speed          │ 155.31 MB/s (20%) │
├───────────────────────────────────┼───────────────────┤
│ AES256 GCM encryption speed       │ 101.22 MB/s (3%)  │
└───────────────────────────────────┴───────────────────┘

che · Apr 5, 2021

m!ka said:

Hello,

I replaced the SSD for ZFS log by "Samsung SM883 240GB 2.5in Data Center SATA 6Gbps"

Code:

Seq. Read    540 MB/s    Seq. Write    480 MB/s
Ran. Read    97K IOPS    Ran. Write    22K IOPS

Not really a big improvement...

Code:

Uploaded 110 chunks in 5 seconds.
Time per request: 46514 microseconds.
TLS speed: 90.17 MB/s
SHA256 speed: 192.15 MB/s
Compression speed: 332.31 MB/s
Decompress speed: 528.12 MB/s
AES256/GCM speed: 101.22 MB/s
Verify speed: 155.31 MB/s
┌───────────────────────────────────┬───────────────────┐
│ Name                              │ Value             │
╞═══════════════════════════════════╪═══════════════════╡
│ TLS (maximal backup upload speed) │ 90.17 MB/s (7%)   │
├───────────────────────────────────┼───────────────────┤
│ SHA256 checksum computation speed │ 192.15 MB/s (10%) │
├───────────────────────────────────┼───────────────────┤
│ ZStd level 1 compression speed    │ 332.31 MB/s (44%) │
├───────────────────────────────────┼───────────────────┤
│ ZStd level 1 decompression speed  │ 528.12 MB/s (44%) │
├───────────────────────────────────┼───────────────────┤
│ Chunk verification speed          │ 155.31 MB/s (20%) │
├───────────────────────────────────┼───────────────────┤
│ AES256 GCM encryption speed       │ 101.22 MB/s (3%)  │
└───────────────────────────────────┴───────────────────┘

A log device will only help improve the write speed for sync writes, it will not help for async writes (which is typically most of them). Here are more details: https://www.servethehome.com/what-is-the-zfs-zil-slog-and-what-makes-a-good-one/

The question is what exactly do you expect to improve? The percent shown in the benchmark data are compared to a rather modern CPU (Ryzen 7 2700X), so you will most certainly not reach those values (see https://pbs.proxmox.com/docs/backup-client.html#benchmarking)

Search

Search

slowness

m!ka

Active Member

tom

Proxmox Staff Member

spirit

Distinguished Member

m!ka

Active Member

tom

Proxmox Staff Member

m!ka

Active Member

tom

Proxmox Staff Member

m!ka

Active Member

m!ka

Active Member

m!ka

Active Member

che

Active Member