Backup Performance - would multithread improve it?

[mohnewald]

Hello,

We run our PBS and have mounted RBD block devices from a Ceph cluster.
The Ceph cluster consists of 12, 14, and 16 enterprise HDDs (not SSDs), making it relatively cheap, large, but somewhat slow in terms of I/O performance.

Benchmarks with parallel threads perform very well.

However, PBS backups seem to be single-threaded.
The benchmark test only measures TCP speed, not the actual write speed to disk, correct?:

root@cluster3-node01:~# proxmox-backup-client benchmark --repository root@pam@proxmox-backup.example.com:Backup-Cluster3
Uploaded 148 chunks in 5 seconds.
Time per request: 35474 microseconds.
TLS speed: 118.23 MB/s
SHA256 speed: 1609.62 MB/s
Compression speed: 390.12 MB/s
Decompress speed: 508.15 MB/s
AES256/GCM speed: 1107.72 MB/s
Verify speed: 384.44 MB/s


Here are my fio benchmarks on my PBS rbd mount:
fio --name=[seq_read/seq_write] --ioengine=libaio --rw=read --bs=1M --numjobs=[1/4] --size=1G --direct=1 --runtime=60 --time_based --filename=/mnt/backup-cluster3/fio-benchmark

Read Threads = 1: [r=87.0MiB/s][r=87 IOPS]
Read Threads = 4: [r=248MiB/s][r=248 IOPS]

Write Theads = 1: [w=17.0MiB/s][w=17 IOPS]
Write Theads = 4: [w=5120KiB/s][w=5 IOPS]


My true restore speed is about 35MB/s tested with a VM (1,25TB in about 10h) in a live enviorment

I know that you can kill the HDDs with little blocks/operation.
I would have expected about 100MB/s. That would make me happy

Questions:

1. is single/multithread my bottleneck here?
2. is there is a multithread option (planned) or something alike for the backup process?

Thanks,
Michael

 

[UdoB]

Write Theads = 1: [w=17.0MiB/s][w=17 IOPS]
Write Theads = 4: [w=5120KiB/s][w=5 IOPS]

That's... unusable. (For PBS.)

1. is single/multithread my bottleneck here?

No. It is a) a network storage and b) rotating rust. Both is not recommended.

https://pbs.proxmox.com/docs/installation.html#recommended-server-system-requirements

 

[mohnewald]

...Easy answer to poor software architecture

if it does not perform, throw hardware at it

Back to my point: The performance "could" be there and maybe there is a way to parallelize it on the architechture level.

 

[Johannes S]

I agree that multi-core support would be great. But that network storage and Rotation rust don't work well is not an Indicator of bad architecture in my book. For the use case of PBS these are readonable expetations imho. Your mileage may varie

 

[tom]

...Easy answer to poor software architecture

no, its the right answer.

HDDs are not recommended for Ceph, Ceph is not recommended for PBS datastore.

PBS is optimized for SSD storage.