Raid 0,1,10 - poor performance

T

TheCosmin

New Member
Sep 13, 2024
12
0
1
Hi All,

I have 3 brand new HPE DL360 Gen11 servers
Dual Xeon Gold 6526Y
512 GB RAM
2 x 480 GB raid1 for Proxmox 9
Raid Controller MR408i-o - 8 x 1.92 TB NVMe SSD Gen4 configured as raid10 for storage.

I have configured this vm config:

1758441672615.png

This VM is running Windows Standard 2025

Below are a few tests with crystaldiskmark

Hardware raid 10 with 8 x disks

1758441734512.png

Hardware raid 1 with 2 x disks

1758441790605.png

Hardware raid 0 with one disk - single disk

1758441811723.png

Hardware raid 0 with 5 x nvme disks

1758441831569.png

It doesn't matter what type of configuration I set on the Raid or the type of disk type or controller I select on the Promox I always get nearly the same speed tests on all 3 nodes.

What are your thoughts on this? have anyone encountered this behavior with HPE servers and MR408i-o controller?
 
TheCosmin said:
Raid Controller MR408i-o - 8 x 1.92 TB NVMe SSD Gen4 configured as raid10 for storage.
Click to expand...
1758445039884.png

As 1 of these disk isn't able to do this should show itself you that the results are useless and so it's even the same to all other benchmark configs.
Use at least a benchmark size of minimum 3x mem of the host which present the disk/volume.
 
Last edited:
spirit said:
It's quite possible that your are cpu limited, as currently vm can only use 1 core for 1 virtual disk.

Multiple threading (with multiple iothread by disk), should be available soon, I have already send patch to the proxmox dev mailing list.
Click to expand...
Thank you for your reply, I have tested with both proxmox v8 and v9 and the results are the same.
 
Well according to the spec sheet the MR408i-o is PCIe Gen4 x8 Lanes, theoretical PCIe max bandwith is 15.754 GB/s, the spec sheet says
Code: 
HPE Compute MR Gen11 Controllers offer up to 13.7 GBs of bandwidth, and eight PCIe Gen4.0 host lanes. HPE Compute MR Controllers offer 3M Random Read IOPS and 240K RAID5 Random Write IOPS.
Depending on your expectations these values are quite good.
 
MarkusKo said:
Well according to the spec sheet the MR408i-o is PCIe Gen4 x8 Lanes, theoretical PCIe max bandwith is 15.754 GB/s, the spec sheet says
Code: 
HPE Compute MR Gen11 Controllers offer up to 13.7 GBs of bandwidth, and eight PCIe Gen4.0 host lanes. HPE Compute MR Controllers offer 3M Random Read IOPS and 240K RAID5 Random Write IOPS.
Depending on your expectations these values are quite good.
Click to expand...
I agree, but when you look at the
RND4K
Q1T1
Values, I can confirm that my home pc without single hdd without raid can do better.
 
Yes, end user SSD's are better with single queue workloads but enterprise ssd's are mostly optimized for higher queue depts. Not sure how the raid controller plays a role in that, maybe you could test/tune the cache settings.
 
MarkusKo said:
Yes, end user SSD's are better with single queue workloads but enterprise ssd's are mostly optimized for higher queue depts. Not sure how the raid controller plays a role in that, maybe you could test/tune the cache settings.
Click to expand...
I have tried every configuration possible to try to gain a little bit more speed and I had no luck.
 
I would not think too much about Crystal Disk Mark performance values in a single VM on a server. Yes they can be an indicator but don't say much about real server workloads.

What file system do you use on top of your raid array for VM storage?
 
Did you already download hpe ssacli raidctrl tool and installed ? What's the configuration of all of your raidctrl as output ?
 
MarkusKo said:
I would not think too much about Crystal Disk Mark performance values in a single VM on a server. Yes they can be an indicator but don't say much about real server workloads.

What file system do you use on top of your raid array for VM storage?
Click to expand...
Windows Std 2025 with SQL Std 2022, sql is not happy about the storage speed.
 
waltar said:
Did you already download hpe ssacli raidctrl tool and installed ? What's the configuration of all of your raidctrl as output ?
Click to expand...
the ssacli will not display anything however the storcli will do it:
/opt/MegaRAID/storcli/storcli64 show
Status Code = 0
Status = Success
Description = None

Number of Controllers = 1
Host Name = pmx-vh-03
Operating System = Linux6.14.11-2-pve

System Overview :
===============

-------------------------------------------------------------------------------
Ctl Model Ports PDs DGs DNOpt VDs VNOpt BBU sPR DS EHS ASOs Hlth
-------------------------------------------------------------------------------
0 HPEMR408i-oGen11 8 8 1 0 1 0 Unkwn On - N 5 NdAtn
-------------------------------------------------------------------------------

Ctl=Controller Index|DGs=Drive groups|VDs=Virtual drives|Fld=Failed
PDs=Physical drives|DNOpt=DG NotOptimal|VNOpt=VD NotOptimal|Opt=Optimal
Msng=Missing|Dgd=Degraded|NdAtn=Need Attention|Unkwn=Unknown
sPR=Scheduled Patrol Read|DS=DimmerSwitch|EHS=Emergency Hot Spare
Y=Yes|N=No|ASOs=Advanced Software Options|BBU=Battery backup unit
Hlth=Health|Safe=Safe-mode boot


/opt/MegaRAID/storcli/storcli64 /c0 show all
Generating detailed summary of the adapter, it may take a while to complete.

Controller = 0
Status = Failure
Description = None

Detailed Status :
===============

----------------------------------------
Ctrl Status ErrCd ErrMsg
----------------------------------------
0 Failed 61 command not supported
----------------------------------------



Basics :
======
Controller = 0
Model = HPE MR408i-o Gen11
Serial Number = SKF0101913
Current Controller Date/Time = 09/21/2025, 13:58:58
Current System Date/time = 09/21/2025, 14:58:58
SAS Address = 500062b22198c2c0
PCI Address = 00:3a:00:00
Mfg Date = 01/06/25
Rework Date = 00/00/00
Revision No = 15009


Version :
=======
Firmware Package Build = 52.32.3-6333
Firmware Version = 5.320.03-4207
NVDATA Version = 5.3200.16-0858
Bios Version = 7.32.00.0_0x07200002
HII Version = 07.32.02.02
Driver Name = megaraid_sas
Driver Version = 07.727.03.00-rc1


Bus :
===
Vendor Id = 0x1000
Device Id = 0x10E2
SubVendor Id = 0x1590
SubDevice Id = 0x3C8
Host Interface = PCIE
Device Interface = SATA-3G
Bus Number = 58
Device Number = 0
Function Number = 0


Pending Images in Flash :
=======================
Image name = No pending images


Status :
======
Controller Status = Needs Attention
Memory Correctable Errors = 0
Memory Uncorrectable Errors = 0
ECC Bucket Count = 0
Any Offline VD Cache Preserved = No
BBU Status = 0
Support PD Firmware Download = No
Lock Key Assigned = No
Failed to get lock key on bootup = No
Lock key has not been backed up = No
Bios was not detected during boot = No
Controller must be rebooted to complete security operation = No
A rollback operation is in progress = No
At least one PFK exists in NVRAM = No
SSC Policy is WB = No
Controller has booted into safe mode = No


Supported Adapter Operations :
============================
Rebuild Rate = Yes
CC Rate = Yes
BGI Rate = Yes
Reconstruct Rate = Yes
Patrol Read Rate = Yes
Alarm Control = No
Cluster Support = No
BBU = Yes
Spanning = Yes
Dedicated Hot Spare = Yes
Revertible Hot Spares = Yes
Foreign Config Import = Yes
Self Diagnostic = Yes
Allow Mixed Redundancy on Array = No
Global Hot Spares = Yes
Deny SCSI Passthrough = No
Deny SMP Passthrough = No
Deny STP Passthrough = No
Support more than 8 Phys = Yes
FW and Event Time in GMT = No
Support Enhanced Foreign Import = Yes
Support Enclosure Enumeration = Yes
Support Allowed Operations = Yes
Abort CC on Error = Yes
Support Multipath = Yes
Support Odd & Even Drive count in RAID1E = No
Support Security = Yes
Support Config Page Model = Yes
Support the OCE without adding drives = Yes
Support EKM = Yes
Snapshot Enabled = No
Support PFK = Yes
Support PI = No
Support Ld BBM Info = No
Support Shield State = Yes
Block SSD Write Disk Cache Change = No
Support Suspend Resume BG ops = Yes
Support Emergency Spares = No
Support Set Link Speed = Yes
Support Boot Time PFK Change = No
Support JBOD = No
Disable Online PFK Change = Yes
Support Perf Tuning = Yes
Support SSD PatrolRead = Yes
Real Time Scheduler = Yes
Support Reset Now = Yes
Support Emulated Drives = Yes
Headless Mode = Yes
Dedicated HotSpares Limited = No
Point In Time Progress = Yes
Extended LD = Yes
Boot Volume Supported = No
Support Uneven span = No
Support Config Auto Balance = No
Support Maintenance Mode = No
Support Diagnostic results = Yes
Support Sesmonitoring = Yes
Support SecurityonJBOD = Yes
Support supportForceTo512e = Yes
Support discardCacheDuringLDDelete = No


Supported PD Operations :
=======================
Force Online = Yes
Force Offline = Yes
Force Rebuild = Yes
Deny Force Failed = No
Deny Force Good/Bad = No
Deny Missing Replace = No
Deny Clear = No
Deny Locate = No
Support Power State = No
Set Power State For Cfg = No
Support T10 Power State = No
Support Temperature = Yes
NCQ = Yes
Support Max Rate SATA = No


Supported VD Operations :
=======================
Read Policy = Yes
Write Policy = Yes
IO Policy = No
Access Policy = Yes
Disk Cache Policy = Yes
Reconstruction = Yes
Deny Locate = No
Deny CC = No
Allow Ctrl Encryption = No
Enable LDBBM = No
Support FastPath = Yes
Performance Metrics = Yes
Power Savings = No
Support Powersave Max With Cache = No
Support Breakmirror = No
Support SSC WriteBack = No
Support SSC Association = No
Support VD Hide = Yes
Support VD Cachebypass = Yes
Support VD discardCacheDuringLDDelete = No


Advanced Software Option :
========================

-----------------------------------------
Adv S/W Opt Time Remaining Mode
-----------------------------------------
MegaRAID FastPath Unlimited -
MegaRAID SafeStore Unlimited -
MegaRAID RAID6 Unlimited -
MegaRAID RAID5 Unlimited -
-----------------------------------------

Safe ID = 3QQJANREVDK771NJEUSN1Q2TVJVQU9M39MU8QIIZ

HwCfg :
=====
ChipRevision = A0
BatteryFRU = N/A
Front End Port Count = 0
Backend Port Count = 8
BBU = Present
Alarm = Missing
Serial Debugger = Present
NVRAM Size = 128KB
Flash Size = 16MB
On Board Memory Size = 4096MB
CacheVault Flash Size = 21.650 GB
TPM = Absent
Upgrade Key = Absent
On Board Expander = Absent
Temperature Sensor for ROC = Present
Temperature Sensor for Controller = Absent
Current Size of CacheCade (GB) = 0
Current Size of FW Cache (MB) = 3142
ROC temperature(Degree Celsius) = 61


Policies :
========

Policies Table :
==============

------------------------------------------------
Policy Current Default
------------------------------------------------
Predictive Fail Poll Interval 60 sec
Interrupt Throttle Active Count 16
Interrupt Throttle Completion 50 us
Rebuild Rate 30 % 30%
PR Rate 30 % 30%
BGI Rate 30 % 30%
Check Consistency Rate 30 % 30%
Reconstruction Rate 30 % 30%
Cache Flush Interval 4s
------------------------------------------------

Flush Time(Default) = 4s
Drive Coercion Mode = none
Auto Rebuild = On
Battery Warning = On
ECC Bucket Size = 15
ECC Bucket Leak Rate (hrs) = 24
Restore HotSpare on Insertion = Off
Expose Enclosure Devices = On
Maintain PD Fail History = Off
Reorder Host Requests = On
Auto detect BackPlane = SGPIO/i2c SEP
Load Balance Mode = Auto
Security Key Assigned = Off
Disable Online Controller Reset = Off
Use drive activity for locate = Off


Boot :
====
BIOS Enumerate VDs = 1
Stop BIOS on Error = Off
Delay during POST = 0
Spin Down Mode = None
Enable Ctrl-R = No
Enable Web BIOS = No
Enable PreBoot CLI = No
Enable BIOS = Yes
Max Drives to Spinup at One Time = 2
Maximum number of direct attached drives to spin up in 1 min = 60
Delay Among Spinup Groups (sec) = 2
Allow Boot with Preserved Cache = Off


High Availability :
=================
Topology Type = None
Cluster Permitted = No
Cluster Active = No


Defaults :
========
Phy Polarity = 0
Phy PolaritySplit = 0
Strip Size = 256kB
Write Policy = WB
Read Policy = None
Cache When BBU Bad = Off
Cached IO = Off
VD PowerSave Policy = Controller Defined
Default spin down time (mins) = 30
Coercion Mode = None
ZCR Config = Unknown
Max Chained Enclosures = 16
Direct PD Mapping = No
Restore Hot Spare on Insertion = No
Expose Enclosure Devices = Yes
Maintain PD Fail History = No
Zero Based Enclosure Enumeration = No
Disable Puncturing = No
EnableLDBBM = No
Un-Certified Hard Disk Drives = Allow
SMART Mode = Mode 6
Enable LED Header = No
LED Show Drive Activity = No
Dirty LED Shows Drive Activity = No
EnableCrashDump = No
Disable Online Controller Reset = No
Treat Single span R1E as R10 = Yes
Power Saving option = Disabled
TTY Log In Flash = Yes
Auto Enhanced Import = Yes
BreakMirror RAID Support = No
Disable Join Mirror = Yes
Enable Shield State = Yes
Time taken to detect CME = 60 sec


Capabilities :
============
Supported Drives = SAS, SATA
Boot Volume Supported = NO
RAID Level Supported = RAID0, RAID1, RAID5, RAID6, RAID10, RAID50,
RAID60, PRL 11, PRL 11 with spanning, SRL 3 supported,
PRL11-RLQ0 DDF layout with no span, PRL11-RLQ0 DDF layout with span
Enable JBOD = No
Mix in Enclosure = Allowed
Mix of SAS/SATA of HDD type in VD = Not Allowed
Mix of SAS/SATA of SSD type in VD = Not Allowed
Mix of SSD/HDD in VD = Not Allowed
SAS Disable = No
Max Arms Per VD = 32
Max Spans Per VD = 8
Max Arrays = 240
Max VD per array = 64
Max Number of VDs = 240
Max Parallel Commands = 5101
Max SGE Count = 60
Max Data Transfer Size = 2048 sectors
Max Strips PerIO = 42
Max Configurable CacheCade Size(GB) = 0
Min Strip Size = 64 KB
Max Strip Size = 1.0 MB


Scheduled Tasks :
===============
Consistency Check Reoccurrence = 168 hrs
Next Consistency check launch = 09/27/2025, 03:00:00
Patrol Read Reoccurrence = 168 hrs
Next Patrol Read launch = 09/27/2025, 03:00:00
Battery learn Reoccurrence = NA
Next Battery Learn = NA
OEMID = LSI

Drive Groups = 1

TOPOLOGY :
========

-------------------------------------------------------------------------
DG Arr Row EID:Slot DID Type State BT Size PDC PI SED DS3 FSpace
-------------------------------------------------------------------------
0 - - - - RAID10 Optl N 6.983 TB enbl N N none N
0 0 - - - RAID1 Optl N 1.745 TB enbl N N none N
0 0 0 252:1 0 DRIVE Onln N 1.745 TB enbl N N none -
0 0 1 252:2 7 DRIVE Onln N 1.745 TB enbl N N none -
0 1 - - - RAID1 Optl N 1.745 TB enbl N N none N
0 1 0 252:3 3 DRIVE Onln N 1.745 TB enbl N N none -
0 1 1 252:4 6 DRIVE Onln N 1.745 TB enbl N N none -
0 2 - - - RAID1 Optl N 1.745 TB enbl N N none N
0 2 0 252:5 2 DRIVE Onln N 1.745 TB enbl N N none -
0 2 1 252:6 5 DRIVE Onln N 1.745 TB enbl N N none -
0 3 - - - RAID1 Optl N 1.745 TB enbl N N none N
0 3 0 252:7 1 DRIVE Onln N 1.745 TB enbl N N none -
0 3 1 252:8 4 DRIVE Onln N 1.745 TB enbl N N none -
-------------------------------------------------------------------------

DG=Disk Group Index|Arr=Array Index|Row=Row Index|EID=Enclosure Device ID
DID=Device ID|Type=Drive Type|Onln=Online|Rbld=Rebuild|Dgrd=Degraded
Pdgd=Partially degraded|Offln=Offline|BT=Background Task Active
PDC=PD Cache|PI=Protection Info|SED=Self Encrypting Drive|Frgn=Foreign
DS3=Dimmer Switch 3|dflt=Default|Msng=Missing|FSpace=Free Space Present

Virtual Drives = 1

VD LIST :
=======

----------------------------------------------------------
DG/VD TYPE State Access Consist Cache sCC Size Name
----------------------------------------------------------
0/239 RAID10 Optl RW No RWBD - 6.983 TB
----------------------------------------------------------

Cac=CacheCade|Rec=Recovery|OfLn=OffLine|Pdgd=Partially Degraded|dgrd=Degraded
Optl=Optimal|RO=Read Only|RW=Read Write|HD=Hidden|B=Blocked|Consist=Consistent|
R=Read Ahead Always|NR=No Read Ahead|WB=WriteBack|
AWB=Always WriteBack|WT=WriteThrough|C=Cached IO|D=Direct IO|sCC=Scheduled
Check Consistency

Physical Drives = 8

PD LIST :
=======

---------------------------------------------------------
EID:Slt DID State DG Size Intf Med SED PI SeSz Sp
---------------------------------------------------------
252:1 0 Onln 0 1.745 TB Unknown SSD N N 512B U
252:2 7 Onln 0 1.745 TB Unknown SSD N N 512B U
252:3 3 Onln 0 1.745 TB Unknown SSD N N 512B U
252:4 6 Onln 0 1.745 TB Unknown SSD N N 512B U
252:5 2 Onln 0 1.745 TB Unknown SSD N N 512B U
252:6 5 Onln 0 1.745 TB Unknown SSD N N 512B U
252:7 1 Onln 0 1.745 TB Unknown SSD N N 512B U
252:8 4 Onln 0 1.745 TB Unknown SSD N N 512B U
---------------------------------------------------------

EID-Enclosure Device ID|Slt-Slot No.|DID-Device ID|DG-DriveGroup
DHS-Dedicated Hot Spare|UGood-Unconfigured Good|GHS-Global Hotspare
UBad-Unconfigured Bad|Onln-Online|Offln-Offline|Intf-Interface
Med-Media Type|SED-Self Encryptive Drive|PI-Protection Info
SeSz-Sector Size|Sp-Spun|U-Up|D-Down|T-Transition|F-Foreign
UGUnsp-Unsupported|UGShld-UnConfigured shielded|HSPShld-Hotspare shielded
CFShld-Configured shielded


BBU_Info :
========

---------------------------
State RetentionTime Temp
---------------------------
Unknown 0 hour(s) 0C
---------------------------
 
TheCosmin said:
It doesn't matter what type of configuration I set on the Raid or the type of disk type or controller I select on the Promox I always get nearly the same speed tests on all 3 nodes.
Click to expand...

Code: 
vm-storage:base-999-disk-1,cache=writeback,iothread=1,size=100G,ssd=1

If you want to check the actual disk speed, shouldn't you disable cache=writeback and benchmark each one separately?

I believe cache=writeback is the reason everything ends up running at similar speeds.

hba9500-16i HUSMR3232ASS200 x1

Default (No Cache)
writeback
 

Attachments

  • CrystalDiskMark_20250921231209.png
    CrystalDiskMark_20250921231209.png
    36.1 KB · Views: 7
  • CrystalDiskMark_20250921224901.png
    CrystalDiskMark_20250921224901.png
    35.8 KB · Views: 6
Last edited:
  • Like
Reactions: waltar
Aah yes, it's a lsi chip based one :-)
/opt/MegaRAID/storcli/storcli64 /call/vall show all
 
waltar said:
Aah yes, it's a lsi chip based one :-)
/opt/MegaRAID/storcli/storcli64 /call/vall show all
Click to expand...
Controller = 0
Status = Success
Description = None


/c0/v239 :
========

----------------------------------------------------------
DG/VD TYPE State Access Consist Cache sCC Size Name
----------------------------------------------------------
0/239 RAID10 Optl RW No RWBD - 6.983 TB
----------------------------------------------------------

Cac=CacheCade|Rec=Recovery|OfLn=OffLine|Pdgd=Partially Degraded|dgrd=Degraded
Optl=Optimal|RO=Read Only|RW=Read Write|HD=Hidden|B=Blocked|Consist=Consistent|
R=Read Ahead Always|NR=No Read Ahead|WB=WriteBack|
AWB=Always WriteBack|WT=WriteThrough|C=Cached IO|D=Direct IO|sCC=Scheduled
Check Consistency


PDs for VD 239 :
==============

---------------------------------------------------------
EID:Slt DID State DG Size Intf Med SED PI SeSz Sp
---------------------------------------------------------
252:1 0 Onln 0 1.745 TB Unknown SSD N N 512B U
252:2 7 Onln 0 1.745 TB Unknown SSD N N 512B U
252:3 3 Onln 0 1.745 TB Unknown SSD N N 512B U
252:4 6 Onln 0 1.745 TB Unknown SSD N N 512B U
252:5 2 Onln 0 1.745 TB Unknown SSD N N 512B U
252:6 5 Onln 0 1.745 TB Unknown SSD N N 512B U
252:7 1 Onln 0 1.745 TB Unknown SSD N N 512B U
252:8 4 Onln 0 1.745 TB Unknown SSD N N 512B U
---------------------------------------------------------

EID-Enclosure Device ID|Slt-Slot No.|DID-Device ID|DG-DriveGroup
DHS-Dedicated Hot Spare|UGood-Unconfigured Good|GHS-Global Hotspare
UBad-Unconfigured Bad|Onln-Online|Offln-Offline|Intf-Interface
Med-Media Type|SED-Self Encryptive Drive|PI-Protection Info
SeSz-Sector Size|Sp-Spun|U-Up|D-Down|T-Transition|F-Foreign
UGUnsp-Unsupported|UGShld-UnConfigured shielded|HSPShld-Hotspare shielded
CFShld-Configured shielded


VD239 Properties :
================
Strip Size = 256 KB
Number of Blocks = 14998568960
VD has Emulated PD = No
Span Depth = 4
Number of Drives Per Span = 2
Write Cache(initial setting) = WriteBack
Disk Cache Policy = Enabled
Encryption = None
Data Protection = None
Active Operations = None
Exposed to OS = Yes
Creation Date = 20-09-2025
Creation Time = 03:16:18 PM
Emulation type = default
Cachebypass size = Cachebypass-64k
Cachebypass Mode = Cachebypass Intelligent
 
uzumo said:
If you want to check the actual disk speed, shouldn't you disable write-back and benchmark each one separately?

I believe writeback is the reason everything ends up running at similar speeds.

hba9500-16i HUSMR3232ASS200 x1

Default (No Cache)
writeback
Click to expand...
1758464829127.png1758464833731.png

Before and after the write back being disabled.
 
If you disable writeback for each of RAID 0, RAID 1, and RAID 10 and benchmark them, there will probably be a difference.

RAID 1 will likely show different results compared to RAID 0 or RAID 10, but I don't think it will deliver performance beyond the current state.

I recall that writeback speed varied depending on the CPU or RAM. I don't know if that's correct.

In my environment, when writeback is enabled, most disks operate at comparable speeds.

https://pve.proxmox.com/wiki/Performance_Tweaks#Disk_Cache
 
Last edited:
Well according to your hardware it's pointless to test RAID0 and RAID1, for VM storage with 8 SSD's you basically have to use RAID10. Using multiple RAID1 would be a special case and that would restrict you in the future (capacity, performance ...) and RAID0 is not really an option ...

Another option for testing would be to switch your RAID controller to HBA mode and use ZFS mirrored vdevs (RAID10 equivalent) but you would have to backup your VM's, destroy your RAID and setup ZFS but hat's another rabbit hole :eek:
 
You must log in or register to reply here.
Top Bottom