slowness

m!ka

Active Member
Jan 21, 2017
16
1
28
Europa
Hello,

I just set up a 1st PBS and I have very poor performance:

Code:
Uploaded 99 chunks in 5 seconds.
Time per request: 51253 microseconds.
TLS speed: 81.83 MB/s
SHA256 speed: 188.74 MB/s
Compression speed: 317.88 MB/s
Decompress speed: 582.48 MB/s
AES256/GCM speed: 112.44 MB/s
Verify speed: 155.37 MB/s
┌───────────────────────────────────┬───────────────────┐
│ Name                              │ Value             │
╞═══════════════════════════════════╪═══════════════════╡
│ TLS (maximal backup upload speed) │ 81.83 MB/s (7%)   │
├───────────────────────────────────┼───────────────────┤
│ SHA256 checksum computation speed │ 188.74 MB/s (9%)  │
├───────────────────────────────────┼───────────────────┤
│ ZStd level 1 compression speed    │ 317.88 MB/s (42%) │
├───────────────────────────────────┼───────────────────┤
│ ZStd level 1 decompression speed  │ 582.48 MB/s (49%) │
├───────────────────────────────────┼───────────────────┤
│ Chunk verification speed          │ 155.37 MB/s (20%) │
├───────────────────────────────────┼───────────────────┤
│ AES256 GCM encryption speed       │ 112.44 MB/s (3%)  │
└───────────────────────────────────┴───────────────────┘

Can you help me?

Hardware:

Code:
  pool: backup
state: ONLINE
config:

    NAME        STATE     READ WRITE CKSUM
    backup      ONLINE       0     0     0
      raidz1-0  ONLINE       0     0     0
        sdc     ONLINE       0     0     0
        sdd     ONLINE       0     0     0
        sde     ONLINE       0     0     0
        sdf     ONLINE       0     0     0
        sdg     ONLINE       0     0     0
    logs  
      sdh       ONLINE       0     0     0

sdc-sdg : 5x SATA Western Digital WD Gold 6 To (WD6003FRYZ-0)
sdh: Crucial SSD M4 CT064M4SSD2 64Go
sda-sdb: Proxmox system (2x Sandisk 128Go SSD mirror-0)

CPU: 2x Intel(R) Xeon(R) CPU E5620 @ 2.40GHz (4 cores, 8 threads)
Memory: 72472Mo DDR3
Motherboard: SuperMicro X8DTN+
Adapter: LSI 6Gbps SAS HBA LSI 9200-8i = (9211-8I) IT Mode
Backplane: BPN-SAS2-826EL1
LAN: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection
 
Last edited:

tom

Proxmox Staff Member
Staff member
Aug 29, 2006
15,659
973
173

spirit

Famous Member
Apr 2, 2010
5,928
716
143
www.odiso.com
aes,tls will be slow, as your cpu don't have hardware support for it.

also, raidz-1 can be slow for write, as you'll have almost the speed of 1 disk, because of parity. (better to do a raid10 with bigger disks, or use a good ssd for zfs log device.
(I'm not sure if your Crucial SSD M4 CT064M4SSD2 64Go is for log device, but you should really use a good datacenter ssd, because this crucial is pretty low for sync write)
 

tom

Proxmox Staff Member
Staff member
Aug 29, 2006
15,659
973
173
500-600Mbit/s.
Your PBS benchmarks do not show these numbers.

The "slowness" is expected for this too old hardware.
 

m!ka

Active Member
Jan 21, 2017
16
1
28
Europa
Your PBS benchmarks do not show these numbers.

The "slowness" is expected for this too old hardware.

TLS (maximal backup upload speed) │ 81.83 MB/s

81.83 MB/s * 8 = 654Mbit/s

Confirmed by "nload":

Capture d’écran 2021-03-29 à 16.52.42.png
 

m!ka

Active Member
Jan 21, 2017
16
1
28
Europa
aes,tls will be slow, as your cpu don't have hardware support for it.

also, raidz-1 can be slow for write, as you'll have almost the speed of 1 disk, because of parity. (better to do a raid10 with bigger disks, or use a good ssd for zfs log device.
(I'm not sure if your Crucial SSD M4 CT064M4SSD2 64Go is for log device, but you should really use a good datacenter ssd, because this crucial is pretty low for sync write

Hello spirit,

I will reconfigure my disks in Raid10 and do some tests.

For the CPU .. are you sure AES is not supported? Intel says this:

Intel® AES New Instructions Yes

For the SSD log, I did not see any improvement with it, and when I monitor with iostat, I do not see any activity.
 

m!ka

Active Member
Jan 21, 2017
16
1
28
Europa
@spirit

indeed, the performance of the Crucial is not very good in writing:

Sustained Sequential Write up to (128k transfer) 95MB/s
Sustained Sequential Read up to (128k transfer) 500MB/s

I read that we recommend a small capacity SSD drive, and I took the 1st small size drive that I found in my stock without paying attention to its performance.

would a Samsung SM883 be better?

Seq. Read 540 MB/s
Seq. Write 480 MB/s

or is the NVME mandatory?

Afterwards, I may be wrong, but via 'iostat' I don't see my Crucial really working ... I wonder if it is useful ...
 

m!ka

Active Member
Jan 21, 2017
16
1
28
Europa
Hello,

I replaced the SSD for ZFS log by "Samsung SM883 240GB 2.5in Data Center SATA 6Gbps"

Code:
Seq. Read    540 MB/s    Seq. Write    480 MB/s
Ran. Read    97K IOPS    Ran. Write    22K IOPS

Not really a big improvement...

Code:
Uploaded 110 chunks in 5 seconds.
Time per request: 46514 microseconds.
TLS speed: 90.17 MB/s
SHA256 speed: 192.15 MB/s
Compression speed: 332.31 MB/s
Decompress speed: 528.12 MB/s
AES256/GCM speed: 101.22 MB/s
Verify speed: 155.31 MB/s
┌───────────────────────────────────┬───────────────────┐
│ Name                              │ Value             │
╞═══════════════════════════════════╪═══════════════════╡
│ TLS (maximal backup upload speed) │ 90.17 MB/s (7%)   │
├───────────────────────────────────┼───────────────────┤
│ SHA256 checksum computation speed │ 192.15 MB/s (10%) │
├───────────────────────────────────┼───────────────────┤
│ ZStd level 1 compression speed    │ 332.31 MB/s (44%) │
├───────────────────────────────────┼───────────────────┤
│ ZStd level 1 decompression speed  │ 528.12 MB/s (44%) │
├───────────────────────────────────┼───────────────────┤
│ Chunk verification speed          │ 155.31 MB/s (20%) │
├───────────────────────────────────┼───────────────────┤
│ AES256 GCM encryption speed       │ 101.22 MB/s (3%)  │
└───────────────────────────────────┴───────────────────┘
 

che

Active Member
Jul 10, 2020
114
26
33
Hello,

I replaced the SSD for ZFS log by "Samsung SM883 240GB 2.5in Data Center SATA 6Gbps"

Code:
Seq. Read    540 MB/s    Seq. Write    480 MB/s
Ran. Read    97K IOPS    Ran. Write    22K IOPS

Not really a big improvement...

Code:
Uploaded 110 chunks in 5 seconds.
Time per request: 46514 microseconds.
TLS speed: 90.17 MB/s
SHA256 speed: 192.15 MB/s
Compression speed: 332.31 MB/s
Decompress speed: 528.12 MB/s
AES256/GCM speed: 101.22 MB/s
Verify speed: 155.31 MB/s
┌───────────────────────────────────┬───────────────────┐
│ Name                              │ Value             │
╞═══════════════════════════════════╪═══════════════════╡
│ TLS (maximal backup upload speed) │ 90.17 MB/s (7%)   │
├───────────────────────────────────┼───────────────────┤
│ SHA256 checksum computation speed │ 192.15 MB/s (10%) │
├───────────────────────────────────┼───────────────────┤
│ ZStd level 1 compression speed    │ 332.31 MB/s (44%) │
├───────────────────────────────────┼───────────────────┤
│ ZStd level 1 decompression speed  │ 528.12 MB/s (44%) │
├───────────────────────────────────┼───────────────────┤
│ Chunk verification speed          │ 155.31 MB/s (20%) │
├───────────────────────────────────┼───────────────────┤
│ AES256 GCM encryption speed       │ 101.22 MB/s (3%)  │
└───────────────────────────────────┴───────────────────┘
A log device will only help improve the write speed for sync writes, it will not help for async writes (which is typically most of them). Here are more details: https://www.servethehome.com/what-is-the-zfs-zil-slog-and-what-makes-a-good-one/

The question is what exactly do you expect to improve? The percent shown in the benchmark data are compared to a rather modern CPU (Ryzen 7 2700X), so you will most certainly not reach those values (see https://pbs.proxmox.com/docs/backup-client.html#benchmarking)
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get your own in 60 seconds.

Buy now!