Very poor storage performance in PVE ceph

[Jordan.zhang]

I tried to test the storage performance of PVE ceph, but the performance I got was very low. I also tried some methods to optimize the test conditions, but there was basically no big change. Any suggestions are appreciated.

hardware configuration:
Node:4
CPU:2 x 6140 18 core 2.3Ghz
MEM:128G
SSD:1x Nvme 800G
HDD:2x 4TB SATA
NIC:4x1Gb+2x10Gb

 

[czechsys]

What about "rados bench" test? Power savings disabled?

 

[justinclift]

Would you be ok to run lsblk -o tran,name,type,size,vendor,model,label,rota,log-sec,phy-sec on one of the Proxmox servers having the problem?

It'll give us a bunch of detailed info about the disks themselves, and save a bunch of back and forth questions.

The standard lsblk output (like you've provided) is missing a bunch of important, useful info.

 

[aaron]

What is low? The bandwidth or IOPS? You did a FIO bench with 4k block size. This way, you will run into IOPS limits, but not bandwidth limits.
Changing the blocksize to, for example, 4M would cause it to rather run into bandwidth limits.

Keep in mind that Ceph will usually issue sync write commands and is very latency sensitive. Therefore, using SSDs with power loss protection (PLP) is recommended.

Make sure that the network works reliably and can achieve the expected speeds, for example with iperf.

 

[Jordan.zhang]

Would you be ok to run lsblk -o tran,name,type,size,vendor,model,label,rota,log-sec,phy-sec on one of the Proxmox servers having the problem?

It'll give us a bunch of detailed info about the disks themselves, and save a bunch of back and forth questions.

The standard lsblk output (like you've provided) is missing a bunch of important, useful info.

Hi Justinclift,

Thanks for your reply, I attached the verbose output of lsblk

 

[justinclift]

Cool. Looking through the model numbers of the pieces, this seems to be your setup:

• Proxmox installed on /dev/sda, which seems like an extremely low performance small SATA flash drive
  • https://www.innodisk.com/en/products/flash-storage/satadom/satadom-sl_3ie3_v2
  • Sequential R/W (MB/sec, max.) 240/160
• Ceph running off three devices
  • 1x 800GB Intel DC P3600 NVMe SSD (data sheet)
  • 2x 4TB 5400RPMToshiba HDDs
    • This seems like a consumer grade low power item?

I'm having trouble finding the data sheet, or really any kind of detailed information on the 4TB Toshiba drives.

There's product info for the 1TB and 2TB drives in that series (here), but nothing definitive for the 4TB one. Can't even tell if they're using CMR or SMR, which is of critical importance performance wise.

IF those 4TB drives use SMR, you'll probably need to go and find some alternative drives that don't.

That being said, I'm not a Ceph guy (not yet anyway), so this is about where my knowledge ends. @aaron (or others) may be able to look at those specs and have helpful suggestions though.

 

[justinclift]

Hmmm, several Amazon reviews (here) are saying that model of Toshiba drive uses SMR technology.

SMR is... to put it bluntly, absolute dog shit for performance. Pretty much worst possible technology.

More info about SMR here, if that helps:

https://arstechnica.com/gadgets/202...-for-sneaking-smr-disks-into-its-nas-channel/

 

[Jordan.zhang]

Hello everyone, after I readjusted the network, I reached what I think is a reasonable range. Thank you for your clues and suggestions.