SSD IOPS Benchmarking

[Trimoon]

Hi There,

I have 2x servers each with 26 SSD's (2 for OS mirror, 24 for Storage), 768GB ram, and Intel(R) Xeon(R) CPU E5-2697 v3 @ 2.60GHz (2 Sockets). I have 3x 9211 HBA cards in IT mode (8 disks each) that should each be able to handle 350k iops.

My end goal is to eventually build a fairly high performing an HA cluster (Haven't figured that out yet as Gluster and Ceph would like 3 nodes and I only have 2... I might just end up building 2 standalone nodes and setup application failover).

Anyways, I'm starting with the storage as a proof of concept and trying to test out what IOPS are possible with SSD's. I have 24 consumer grade (Ill buy enterprise grade when i can prove they will work this is just for testing) SSD's that are all the same model.

Tried to follow the PDF here https://www.proxmox.com/en/downloads/item/proxmox-ve-zfs-benchmark-2020 but interestingly it doesnt matter if i have 1 or 24 SSD's in a stripe, mirror, raid 10 or raidz, i always seem to get 70,000 iops give or take within a windows vm running diskspd. If i use FIO on proxmox VE for the zpool i get even worse performance, about 30k iops

I understand that with zfs raidz you only get the performance of 1 drive so that makes sense, but when I stripe or mirror or raid 10 everything shouldn't i be getting more iops than 1 drive's worth?


Can someone point me in the right direction?

Much appreciated

 

[LnxBil]

I understand that with zfs raidz you only get the performance of 1 drive so that makes sense, but when I stripe or mirror or raid 10 everything shouldn't i be getting more iops than 1 drive's worth?

Yes and it scales linearly with the queue depth. One single thread benchmark will not be fast.

Can someone point me in the right direction?

Don't benchmark consumer grade SSDs ...

 

[Trimoon]

Yes and it scales linearly with the queue depth. One single thread benchmark will not be fast.


Don't benchmark consumer grade SSDs ...

Thanks. You pointed me in the right direction with queue depth. I was following https://pve.proxmox.com/wiki/Benchmarking_Storage but iodepth is what i was missing. Tests now pull 200k iops instead of 70 using:

fio --ioengine=libaio --direct=1 --sync=1 --rw=read --bs=4k --numjobs=1 --iodepth=32 --name seq_read --size=1G --filename=/RAIDZ2/test

This is good, but still doesnt seem to be a good test because im getting around 200k iops no matter what. It doesnt matter if i use SAS / SATA / SSD, RAIDZ / Stripe, Consumer or Enterprise drives they all produce about the same iops.

The only other way i can think of comparing storage is creating making 2 VM's and running iPerf3 tests between them.

I feel like there has to be a better way of benchmarking storage pools for Proxmox HV at the hypervisor level no?

 

[LnxBil]

fio --ioengine=libaio --direct=1 --sync=1 --rw=read --bs=4k --numjobs=1 --iodepth=32 --name seq_read --size=1G --filename=/RAIDZ2/test

Why sequential read? IOPS matter in sync random read/write. You're benchmarking the only thing, that SSDs perform not much better than harddisks... sequential read is THE best case for harddisks and therefore the difference between harddisks and SSDs is the smallest.

 

[Trimoon]

Why sequential read? IOPS matter in sync random read/write. You're benchmarking the only thing, that SSDs perform not much better than harddisks... sequential read is THE best case for harddisks and therefore the difference between harddisks and SSDs is the smallest.

Got it. I ended up using:

fio --ioengine=libaio --direct=1 --rw=randrw --bs=4k --numjobs=4 --iodepth=64 --name benchmark_test --size=5GB -runtime=120 --filename=/<zpoolname>/test --group_reporting

now i see a difference. 1690 vs 3935 avg iops and 81 vs 177210 usec in 99.00th clat percentiles.

Thank you!