Proxmox shared storage low performance

[martyz]

Hi all,
please could someone explain me what might be wrong with our new storage to perform really badly? We are slowly moving from VMware to PVE and also from old EMC VNXe3200 to 2x SuperMicro storage servers based on X10SRi-F-O with SLES HA. Those SuperMicro servers are not in production yet and testing is not optimistic so far. We are using NFS for shared storages and here are numbers:

#pveperf on NVMe RAID0 over NFS
CPU BOGOMIPS:      211209.12
REGEX/SECOND:      2172544
HD SIZE:           744.86 GB (proxmox01:/nvme)
FSYNCS/SECOND:     2669.72
DNS EXT:           25.32 ms
DNS INT:           0.88 ms

#pveperf on WD-RED RAID5 over NFS
CPU BOGOMIPS:      211209.12
REGEX/SECOND:      2242008
HD SIZE:           5566.31 GB (proxmox03:/wdred)
FSYNCS/SECOND:     1381.81
DNS EXT:           26.93 ms
DNS INT:           0.88 ms

#pveperf on TOSHIBA MG04ACA400E RAID10 over NFS
CPU BOGOMIPS:      211209.12
REGEX/SECOND:      2120654
HD SIZE:           502.96 GB (SLESHAstorage:/exports)
FSYNCS/SECOND:     24.65
DNS EXT:           25.98 ms
DNS INT:           0.90 ms

and I have no clue what could I do to increase FSYNCS on SLES HA storage - even PC grade WD RED RAID5 with onboard Intel or Marvell SATA controller is outperforming our enterprise grade SuperMicro HA storage with LSI SAS3008 controller...

I have already tried tuning TCP, NFS params, ext4/XFS mount opts, BIOS and controller (not much I could do there) and nothing... Moving 80GB qcow2 disk took 12 minutes when moving from NVMe to SLES HAstorage, 5 minutes when moving from NVMe to EMC and 4 minutes when moving from SLES HAstorage to NVMe. So reading is not an issue, writing with rsync 80GB file to SLES HAstorage also flies (311.13MB/s, 4m23s) but disk moves + FSYNCS scares me a lot. I would be glad for any hint. Thank you.

MartyZ

 

[LnxBil]

So reading is not an issue, writing with rsync 80GB file to SLES HAstorage also flies (311.13MB/s, 4m23s) but disk moves + FSYNCS scares me a lot.

rsync is not a good sync write benchmark. Test your storage with a good storage benchmark like fio with sync writes. It it's also slow, contact the SLES HAstorage support channel.

 

[martyz]

rsync is not a good sync write benchmark. Test your storage with a good storage benchmark like fio with sync writes. It it's also slow, contact the SLES HAstorage support channel.

fio is what I did first - average values were above EMC, but it is synthetic - real use scenario seems to be worse and I have no idea what is bottleneck ...

 

[LnxBil]

real use scenario seems to be worse and I have no idea what is bottleneck ...

The last resort is always dtrace and see deeper into your path down to the storage.

 

[martyz]

The last resort is always dtrace and see deeper into your path down to the storage.

That is kinda out of my scale Could you please give me some hint how can I dtrace slow storage and how to interpret results? Or what fio command is closest to pveperf fsyncs so I can compare it locally and through NFS and if results differs significantly I can complain at SLES? Thank you.

 

[LnxBil]

That is kinda out of my scale Could you please give me some hint how can I dtrace slow storage and how to interpret results?

Sorry, I haven't done it either.

Or what fio command is closest to pveperf fsyncs so I can compare it locally and through NFS and if results differs significantly I can complain at SLES? Thank you.

As long as you use the same fio command, it does not matter what you use. A good starting point is sequential read. You can then just change the --rw=read parameter to write, randread, randwrite and randrw.

 

[martyz]

ok thanks a lot @LnxBil we will try some better controller with huge cache (probably Areca 1883) - our LSI SAS3008 is without cache - and Areca says "For Areca raid adapters, write cache works on all configured drives including array members and passthrough drives." so this might help to boost our fsyncs - if not, then SLES support ticket will be my next step