extremely slow LSI 8265-8i system

T

tarball

Member
Nov 1, 2011
24
1
23
Hi,

I do have an H8SGL based box with a 6328 CPU; 32GB RAM an LSI 9265-8i raid card; connecting to 8 sata2 drives hooked up to a SM SAS 743TQ backplane; raid-0 (Always writeback, cached I/O, ..).
Everything is flashed to the latest BIOS/firmware.
I've tried just about anything, performance remains abysmal for buffered reads; the box also seems to hang randomly for random processes; takes me 10s to mount a drive the first time for example; or a simple ping takes 5s to fire. The raid controller claims everything is fine. I tried disabling aspm and msi; no luck. Any ideas ? I tried the latest Debian 3.x kernel -same issues.


Raid array (6 disks raid0, ext4):
#pveperf /data
CPU BOGOMIPS: 76794.24
REGEX/SECOND: 937431
HD SIZE: 2015.86 GB (/dev/sda1)
BUFFERED READS: 0.43 MB/sec
AVERAGE SEEK TIME: 809.43 ms
FSYNCS/SECOND: 5821.39
DNS EXT: 16.23 ms
DNS INT: 17.62 ms

HDPARM:
root@wtf:~# hdparm -Tt /dev/sda


/dev/sda:
Timing cached reads: 6752 MB in 2.00 seconds = 3377.81 MB/sec
Timing buffered disk reads: 10 MB in 11.13 seconds = 920.07 kB/sec

raid array: 2 disks (raid0)

 #pveperf /data2
CPU BOGOMIPS: 76794.24
REGEX/SECOND: 907240
HD SIZE: 915.90 GB (/dev/sdb1)
BUFFERED READS: 0.54 MB/sec
AVERAGE SEEK TIME: 559.91 ms
FSYNCS/SECOND: 1697.49
DNS EXT: 15.99 ms
DNS INT: 17.50 ms

hdparm:
/dev/sdb:
Timing cached reads: 1634 MB in 2.00 seconds = 816.98 MB/sec
Timing buffered disk reads: 4 MB in 7.49 seconds = 547.00 kB/sec

SINGLE SSD on internal AMD SATA:

#pveperf .
CPU BOGOMIPS: 76794.24
REGEX/SECOND: 915477
HD SIZE: 94.49 GB (/dev/mapper/pve-root)
BUFFERED READS: 189.18 MB/sec
AVERAGE SEEK TIME: 0.07 ms
FSYNCS/SECOND: 3051.25
DNS EXT: 16.70 ms
DNS INT: 51.98 ms
hdparm:
#hdparm -Tt /dev/sdc
/dev/sdc:
Timing cached reads: 6762 MB in 2.00 seconds = 3382.82 MB/sec
Timing buffered disk reads: 572 MB in 3.01 seconds = 190.00 MB/sec


pveversion -v

proxmox-ve-2.6.32: 3.2-129 (running kernel: 2.6.32-30-pve)
pve-manager: 3.2-4 (running version: 3.2-4/e24a91c1)
pve-kernel-2.6.32-30-pve: 2.6.32-130
pve-kernel-2.6.32-29-pve: 2.6.32-126
lvm2: 2.02.98-pve4
clvm: 2.02.98-pve4
corosync-pve: 1.4.5-1
openais-pve: 1.1.4-3
libqb0: 0.11.1-2
redhat-cluster-pve: 3.2.0-2
resource-agents-pve: 3.9.2-4
fence-agents-pve: 4.0.5-1
pve-cluster: 3.0-12
qemu-server: 3.1-16
pve-firmware: 1.1-3
libpve-common-perl: 3.0-18
libpve-access-control: 3.0-11
libpve-storage-perl: 3.0-19
pve-libspice-server1: 0.12.4-3
vncterm: 1.1-6
vzctl: 4.0-1pve5
vzprocps: 2.0.11-2
vzquota: 3.1-2
pve-qemu-kvm: 1.7-8
ksm-control-daemon: 1.1-1
glusterfs-client: 3.4.2-1
 
tarball said:
Hi,

I do have an H8SGL based box with a 6328 CPU; 32GB RAM an LSI 9265-8i raid card; connecting to 8 sata2 drives hooked up to a SM SAS 743TQ backplane; raid-0 (Always writeback, cached I/O, ..).
Everything is flashed to the latest BIOS/firmware.
I've tried just about anything, performance remains abysmal for buffered reads; the box also seems to hang randomly for random processes; takes me 10s to mount a drive the first time for example; or a simple ping takes 5s to fire. The raid controller claims everything is fine. I tried disabling aspm and msi; no luck. Any ideas ? I tried the latest Debian 3.x kernel -same issues.
...
Click to expand...
Hi,
sounds like an defect disk or that the controller has an problem with your backplane.
Can you connect your drives directly to the controller (with SF8087 to 4*Sata Cable)?

BTW. do you think that an raid-0 on 8 disks is an good idea? I would go for raid-10 to avoid data loss.

Udo
 
Thanks Udo - yes, that was my next move. I'll report as soon as this is done.

The raid level is currently set to raid0 because it doesn't require any build time for me to test; it is also the one that should show the best performance when tesing. I have tried raid5,6 - same issue.

udo said:
Hi,
sounds like an defect disk or that the controller has an problem with your backplane.
Can you connect your drives directly to the controller (with SF8087 to 4*Sata Cable)?

BTW. do you think that an raid-0 on 8 disks is an good idea? I would go for raid-10 to avoid data loss.

Udo
Click to expand...
 
Removed the drives from the backplane and directly connected them to the controller via a SF8087 to 4*Sata Cable . Same story..

 hdparm -Tt /dev/sda


/dev/sda:
Timing cached reads: 6720 MB in 2.00 seconds = 3362.02 MB/sec
Timing buffered disk reads: 4 MB in 7.31 seconds = 560.30 kB/sec

(I also tested every drive individually; quick SMART 2' test). They're all good.
Is there some compatibility issue between that card and the motherboard/chipset maybe ? It's fairly standard hardware i think.
 
tarball said:
Removed the drives from the backplane and directly connected them to the controller via a SF8087 to 4*Sata Cable . Same story..

 hdparm -Tt /dev/sda


/dev/sda:
Timing cached reads: 6720 MB in 2.00 seconds = 3362.02 MB/sec
Timing buffered disk reads: 4 MB in 7.31 seconds = 560.30 kB/sec

(I also tested every drive individually; quick SMART 2' test). They're all good.
Is there some compatibility issue between that card and the motherboard/chipset maybe ? It's fairly standard hardware i think.
Click to expand...


maybe can you test with pve-kernel 3.10, to be sure it's not a driver bug ?
 
spirit said:
maybe can you test with pve-kernel 3.10, to be sure it's not a driver bug ?
Click to expand...

I tried ..same issue - inconsistent results for a server that does *nothing*;

Linux wtf 3.10.0-3-pve #1 SMP Thu Jun 12 13:50:49 CEST 2014 x86_64 GNU/Linux

/dev/sda:
Timing cached reads: 7116 MB in 2.00 seconds = 3560.52 MB/sec
Timing buffered disk reads: 886 MB in 3.00 seconds = 294.89 MB/sec

hdparm -Tt /dev/sda


/dev/sda:
Timing cached reads: 6998 MB in 2.00 seconds = 3502.09 MB/sec
Timing buffered disk reads: 402 MB in 3.00 seconds = 133.84 MB/sec

# hdparm -Tt /dev/sda


/dev/sda:
Timing cached reads: 6920 MB in 2.00 seconds = 3462.75 MB/sec
Timing buffered disk reads: 1192 MB in 3.00 seconds = 397.26 MB/sec


# hdparm -Tt /dev/sda


/dev/sda:
Timing cached reads: 6898 MB in 2.00 seconds = 3451.92 MB/sec
Timing buffered disk reads: 16 MB in 3.24 seconds = 4.94 MB/sec

pveperf .
CPU BOGOMIPS: 76799.52
REGEX/SECOND: 759303
HD SIZE: 2015.74 GB (/dev/sda1)
BUFFERED READS: 79.32 MB/sec
AVERAGE SEEK TIME: 22.57 ms
FSYNCS/SECOND: 4706.28
DNS EXT: 17.94 ms
DNS INT: 16.87 ms


pveperf .
CPU BOGOMIPS: 76799.52
REGEX/SECOND: 773683
HD SIZE: 2015.74 GB (/dev/sda1)
BUFFERED READS: 340.34 MB/sec
AVERAGE SEEK TIME: 18.57 ms
FSYNCS/SECOND: 4941.23
DNS EXT: 17.06 ms
DNS INT: 16.84 ms

pveperf .
CPU BOGOMIPS: 76799.52
REGEX/SECOND: 748236
HD SIZE: 2015.74 GB (/dev/sda1)
BUFFERED READS: 0.49 MB/sec
AVERAGE SEEK TIME: 21.30 ms
FSYNCS/SECOND: 4868.97
DNS EXT: 132.99 ms
DNS INT: 214.66 ms
 
tarball said:
I tried ..same issue - inconsistent results for a server that does *nothing*;

Linux wtf 3.10.0-3-pve #1 SMP Thu Jun 12 13:50:49 CEST 2014 x86_64 GNU/Linux

/dev/sda:
Timing cached reads: 7116 MB in 2.00 seconds = 3560.52 MB/sec
Timing buffered disk reads: 886 MB in 3.00 seconds = 294.89 MB/sec

hdparm -Tt /dev/sda


/dev/sda:
Timing cached reads: 6998 MB in 2.00 seconds = 3502.09 MB/sec
Timing buffered disk reads: 402 MB in 3.00 seconds = 133.84 MB/sec

# hdparm -Tt /dev/sda


/dev/sda:
Timing cached reads: 6920 MB in 2.00 seconds = 3462.75 MB/sec
Timing buffered disk reads: 1192 MB in 3.00 seconds = 397.26 MB/sec


# hdparm -Tt /dev/sda


/dev/sda:
Timing cached reads: 6898 MB in 2.00 seconds = 3451.92 MB/sec
Timing buffered disk reads: 16 MB in 3.24 seconds = 4.94 MB/sec

pveperf .
CPU BOGOMIPS: 76799.52
REGEX/SECOND: 759303
HD SIZE: 2015.74 GB (/dev/sda1)
BUFFERED READS: 79.32 MB/sec
AVERAGE SEEK TIME: 22.57 ms
FSYNCS/SECOND: 4706.28
DNS EXT: 17.94 ms
DNS INT: 16.87 ms


pveperf .
CPU BOGOMIPS: 76799.52
REGEX/SECOND: 773683
HD SIZE: 2015.74 GB (/dev/sda1)
BUFFERED READS: 340.34 MB/sec
AVERAGE SEEK TIME: 18.57 ms
FSYNCS/SECOND: 4941.23
DNS EXT: 17.06 ms
DNS INT: 16.84 ms

pveperf .
CPU BOGOMIPS: 76799.52
REGEX/SECOND: 748236
HD SIZE: 2015.74 GB (/dev/sda1)
BUFFERED READS: 0.49 MB/sec
AVERAGE SEEK TIME: 21.30 ms
FSYNCS/SECOND: 4868.97
DNS EXT: 132.99 ms
DNS INT: 214.66 ms
Click to expand...


I just recreated 2 arrays raid-0; 4 heads per array - every possible cache turned on...

hdparm -Tt /dev/sda


/dev/sda:
Timing cached reads: 6862 MB in 2.00 seconds = 3433.56 MB/sec
Timing buffered disk reads: 12 MB in 11.14 seconds = 1.08 MB/sec
root@royale:~# hdparm -Tt /dev/sdb


/dev/sdb:
Timing cached reads: 6890 MB in 2.00 seconds = 3448.23 MB/sec
Timing buffered disk reads: 10 MB in 11.34 seconds = 902.63 kB/sec


Card config:

Adapter #0


==============================================================================
Versions
================
Product Name : LSI MegaRAID SAS 9265-8i
Serial No : SV12502299
FW Package Build: 23.28.0-0015


Mfg. Data
================
Mfg. Date : 06/12/11
Rework Date : 00/00/00
Revision No : 01A
Battery FRU : N/A


Image Versions in Flash:
================
BIOS Version : 5.46.02.1_4.16.08.00_0x06060A03
WebBIOS Version : 6.1-72-e_72-01-Rel
Preboot CLI Version: 05.07-00:#%00011
FW Version : 3.400.45-3507
NVDATA Version : 2.1403.03-0133
Boot Block Version : 2.05.00.00-0010
BOOT Version : 07.26.26.219


Pending Images in Flash
================
None


PCI Info
================
Controller Id : 0000
Vendor Id : 1000
Device Id : 005b
SubVendorId : 1000
SubDeviceId : 9265


Host Interface : PCIE


ChipRevision : B0


Link Speed : 0
Number of Frontend Port: 0
Device Interface : PCIE


Number of Backend Port: 8
Port : Address
0 4433221100000000
1 4433221101000000
2 4433221102000000
3 4433221103000000
4 4433221105000000
5 4433221106000000
6 4433221107000000
7 4433221104000000


HW Configuration
================
SAS Address : 500605b00385ae30
BBU : Present
Alarm : Present
NVRAM : Present
Serial Debugger : Present
Memory : Present
Flash : Present
Memory Size : 1024MB
TPM : Absent
On board Expander: Absent
Upgrade Key : Absent
Temperature sensor for ROC : Present
Temperature sensor for controller : Absent


ROC temperature : 76 degree Celsius


Settings
================
Current Time : 22:25:19 7/7, 2014
Predictive Fail Poll Interval : 300sec
Interrupt Throttle Active Count : 16
Interrupt Throttle Completion : 50us
Rebuild Rate : 30%
PR Rate : 5%
BGI Rate : 30%
Check Consistency Rate : 30%
Reconstruction Rate : 30%
Cache Flush Interval : 4s
Max Drives to Spinup at One Time : 2
Delay Among Spinup Groups : 4s
Physical Drive Coercion Mode : Disabled
Cluster Mode : Disabled
Alarm : Enabled
Auto Rebuild : Enabled
Battery Warning : Enabled
Ecc Bucket Size : 15
Ecc Bucket Leak Rate : 1440 Minutes
Restore HotSpare on Insertion : Disabled
Expose Enclosure Devices : Enabled
Maintain PD Fail History : Enabled
Host Request Reordering : Enabled
Auto Detect BackPlane Enabled : SGPIO/i2c SEP
Load Balance Mode : Auto
Use FDE Only : No
Security Key Assigned : No
Security Key Failed : No
Security Key Not Backedup : No
Default LD PowerSave Policy : Controller Defined
Maximum number of direct attached drives to spin up in 1 min : 30
Auto Enhanced Import : No
Any Offline VD Cache Preserved : No
Allow Boot with Preserved Cache : No
Disable Online Controller Reset : No
PFK in NVRAM : No
Use disk activity for locate : Yes
POST delay : 90 seconds
BIOS Error Handling : Ignore Errors
Current Boot Mode :Normal
Capabilities
================
RAID Level Supported : RAID0, RAID1, RAID5, RAID6, RAID00, RAID10, RAID50, RAID60, PRL 11, PR L 11 with spanning, SRL 3 supported, PRL11-RLQ0 DDF layout with no span, PRL11-RLQ0 DDF layout with span
Supported Drives : SAS, SATA


Allowed Mixing:


Mix in Enclosure Allowed
Mix of SAS/SATA of HDD type in VD Allowed


Status
================
ECC Bucket Count : 0


Limitations
================
Max Arms Per VD : 32
Max Spans Per VD : 8
Max Arrays : 128
Max Number of VDs : 64
Max Parallel Commands : 1008
Max SGE Count : 60
Max Data Transfer Size : 8192 sectors
Max Strips PerIO : 42
Max LD per array : 16
Min Strip Size : 8 KB
Max Strip Size : 1.0 MB
Max Configurable CacheCade Size: 0 GB
Current Size of CacheCade : 0 GB
Current Size of FW Cache : 875 MB


Device Present
================
Virtual Drives : 2
Degraded : 0
Offline : 0
Physical Devices : 9
Disks : 8
Critical Disks : 0
Failed Disks : 0


Supported Adapter Operations
================
Rebuild Rate : Yes
CC Rate : Yes
BGI Rate : Yes
Reconstruct Rate : Yes
Patrol Read Rate : Yes
Alarm Control : Yes
Cluster Support : No
BBU : Yes
Spanning : Yes
Dedicated Hot Spare : Yes
Revertible Hot Spares : Yes
Foreign Config Import : Yes
Self Diagnostic : Yes
Allow Mixed Redundancy on Array : No
Global Hot Spares : Yes
Deny SCSI Passthrough : No
Deny SMP Passthrough : No
Deny STP Passthrough : No
Support Security : No
Snapshot Enabled : No
Support the OCE without adding drives : Yes
Support PFK : Yes
Support PI : Yes
Support Boot Time PFK Change : No
Disable Online PFK Change : No
Support LDPI Type1 : No
Support LDPI Type2 : No
Support LDPI Type3 : No
PFK TrailTime Remaining : 0 days 0 hours
Support Shield State : Yes
Block SSD Write Disk Cache Change: Yes
Support Online FW Update : Yes


Supported VD Operations
================
Read Policy : Yes
Write Policy : Yes
IO Policy : Yes
Access Policy : Yes
Disk Cache Policy : Yes
Reconstruction : Yes
Deny Locate : No
Deny CC : No
Allow Ctrl Encryption: No
Enable LDBBM : No
Support Breakmirror : No
Power Savings : No


Supported PD Operations
================
Force Online : Yes
Force Offline : Yes
Force Rebuild : Yes
Deny Force Failed : No
Deny Force Good/Bad : No
Deny Missing Replace : No
Deny Clear : No
Deny Locate : No
Support Temperature : Yes
NCQ : Yes
Disable Copyback : No
Enable JBOD : No
Enable Copyback on SMART : No
Enable Copyback to SSD on SMART Error : Yes
Enable SSD Patrol Read : No
PR Correct Unconfigured Areas : Yes
Enable Spin Down of UnConfigured Drives : No
Disable Spin Down of hot spares : Yes
Spin Down time : 30
T10 Power State : No
Error Counters
================
Memory Correctable Errors : 0
Memory Uncorrectable Errors : 0


Cluster Information
================
Cluster Permitted : No
Cluster Active : No


Default Settings
================
Phy Polarity : 0
Phy PolaritySplit : 0
Background Rate : 30
Strip Size : 256kB
Flush Time : 4 seconds
Write Policy : WB
Read Policy : Adaptive
Cache When BBU Bad : Disabled
Cached IO : No
SMART Mode : Mode 6
Alarm Disable : Yes
Coercion Mode : None
ZCR Config : Unknown
Dirty LED Shows Drive Activity : No
BIOS Continue on Error : 0
Spin Down Mode : None
Allowed Device Type : SAS/SATA Mix
Allow Mix in Enclosure : Yes
Allow HDD SAS/SATA Mix in VD : Yes
Allow SSD SAS/SATA Mix in VD : No
Allow HDD/SSD Mix in VD : No
Allow SATA in Cluster : No
Max Chained Enclosures : 16
Disable Ctrl-R : Yes
Enable Web BIOS : Yes
Direct PD Mapping : No
BIOS Enumerate VDs : Yes
Restore Hot Spare on Insertion : No
Expose Enclosure Devices : Yes
Maintain PD Fail History : Yes
Disable Puncturing : No
Zero Based Enclosure Enumeration : No
PreBoot CLI Enabled : Yes
LED Show Drive Activity : Yes
Cluster Disable : Yes
SAS Disable : No
Auto Detect BackPlane Enable : SGPIO/i2c SEP
Use FDE Only : No
Enable Led Header : No
Delay during POST : 0
EnableCrashDump : No
Disable Online Controller Reset : No
EnableLDBBM : No
Un-Certified Hard Disk Drives : Allow
Treat Single span R1E as R10 : No
Max LD per array : 16
Power Saving option : Don't Auto spin down Configured Drives
Max power savings option is not allowed for LDs. Only T10 power conditions are to be used.
Default spin down time in minutes: 30
Enable JBOD : No
TTY Log In Flash : No
Auto Enhanced Import : No
BreakMirror RAID Support : No
Disable Join Mirror : No
Enable Shield State : Yes
Time taken to detect CME : 60s


Exit Code: 0x00
 
Last edited:
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back