I have a Broadcom / LSI MegaRAID SAS-3 3008 to which are 7 disk connected. All of them are configured in BIOS to be in JBOD mode, so that the operating system could see them as raw disks.
The problem is, that the disk are only visible via lsblk on OpenSuse. I tested proxmox and ubuntu and, both of them did not see the disks.
Are there some weird proprietary drivers that are only default loaded on OpenSuse or something?
Here is the lspci output.
Opensuse
proxmox:
When compared, one of the notable changes is that proxmox is missing the line: Kernel driver in use: megaraid_sas
But the easy approach does not work:
```
```
The dmsg looks like this
I reinstalled the operating systems about 6 times and every time the same problem occurs. any ideas how I could make these disks available under proxmox.
The problem is, that the disk are only visible via lsblk on OpenSuse. I tested proxmox and ubuntu and, both of them did not see the disks.
Are there some weird proprietary drivers that are only default loaded on OpenSuse or something?
Here is the lspci output.
Opensuse
Code:
01:00.0 RAID bus controller: Broadcom / LSI MegaRAID SAS-3 3008 [Fury] (rev 02)
Subsystem: Fujitsu Technology Solutions PRAID CP400i [D3307-A12]
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0
Interrupt: pin A routed to IRQ 16
Region 0: I/O ports at e000 [size=256]
Region 1: Memory at dfb00000 (64-bit, non-prefetchable) [size=64K]
Region 3: Memory at dfa00000 (64-bit, non-prefetchable) [size=1M]
Expansion ROM at df900000 [disabled] [size=1M]
Capabilities: [50] Power Management version 3
Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [68] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 4096 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us
ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset- SlotPowerLimit 0.000W
DevCtl: Report errors: Correctable+ Non-Fatal+ Fatal+ Unsupported-
RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+
MaxPayload 256 bytes, MaxReadReq 512 bytes
DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr- TransPend-
LnkCap: Port #0, Speed 8GT/s, Width x8, ASPM L0s, Exit Latency L0s <2us
ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 8GT/s, Width x8, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range BC, TimeoutDis+, LTR-, OBFF Not Supported
AtomicOpsCap: 32bit- 64bit- 128bitCAS-
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
AtomicOpsCtl: ReqEn-
LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete+, EqualizationPhase1+
EqualizationPhase2+, EqualizationPhase3+, LinkEqualizationRequest-
Capabilities: [a8] MSI: Enable- Count=1/1 Maskable+ 64bit+
Address: 0000000000000000 Data: 0000
Masking: 00000000 Pending: 00000000
Capabilities: [c0] MSI-X: Enable+ Count=97 Masked-
Vector table: BAR=1 offset=0000e000
PBA: BAR=1 offset=0000f000
Capabilities: [100 v2] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
AERCap: First Error Pointer: 00, ECRCGenCap- ECRCGenEn- ECRCChkCap- ECRCChkEn-
MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
HeaderLog: 04000001 0000000f 20010000 5f2e8f6f
Capabilities: [1e0 v1] #19
Capabilities: [1c0 v1] Power Budgeting <?>
Capabilities: [148 v1] Alternative Routing-ID Interpretation (ARI)
ARICap: MFVC- ACS-, Next Function: 0
ARICtl: MFVC- ACS-, Function Group: 0
Kernel driver in use: megaraid_sas
Kernel modules: megaraid_sas
Code:
01:00.0 RAID bus controller: Broadcom / LSI MegaRAID SAS-3 3008 [Fury] (rev 02)
Subsystem: Fujitsu Technology Solutions PRAID CP400i [D3307-A12]
Control: I/O+ Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort+ >SERR- <PERR- INTx-
Interrupt: pin A routed to IRQ 16
IOMMU group: 1
Region 0: I/O ports at e000 [size=256]
Region 1: Memory at dfb00000 (64-bit, non-prefetchable) [size=64K]
Region 3: Memory at dfa00000 (64-bit, non-prefetchable) [size=1M]
Expansion ROM at df900000 [disabled] [size=1M]
Capabilities: [50] Power Management version 3
Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [68] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 4096 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us
ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset- SlotPowerLimit 0W
DevCtl: CorrErr+ NonFatalErr+ FatalErr+ UnsupReq-
RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+
MaxPayload 256 bytes, MaxReadReq 512 bytes
DevSta: CorrErr+ NonFatalErr- FatalErr- UnsupReq+ AuxPwr- TransPend-
LnkCap: Port #0, Speed 8GT/s, Width x8, ASPM L0s, Exit Latency L0s <2us
ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 8GT/s, Width x8
TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range BC, TimeoutDis+ NROPrPrP- LTR-
10BitTagComp- 10BitTagReq- OBFF Not Supported, ExtFmt- EETLPPrefix-
EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
FRS- TPHComp- ExtTPHComp-
AtomicOpsCap: 32bit- 64bit- 128bitCAS-
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR- 10BitTagReq- OBFF Disabled,
AtomicOpsCtl: ReqEn-
LnkCap2: Supported Link Speeds: 2.5-8GT/s, Crosslink- Retimer- 2Retimers- DRS-
LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
Compliance Preset/De-emphasis: -6dB de-emphasis, 0dB preshoot
LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete+ EqualizationPhase1+
EqualizationPhase2+ EqualizationPhase3+ LinkEqualizationRequest-
Retimer- 2Retimers- CrosslinkRes: unsupported
Capabilities: [a8] MSI: Enable- Count=1/1 Maskable+ 64bit+
Address: 0000000000000000 Data: 0000
Masking: 00000000 Pending: 00000000
Capabilities: [c0] MSI-X: Enable- Count=97 Masked-
Vector table: BAR=1 offset=0000e000
PBA: BAR=1 offset=0000f000
Capabilities: [100 v2] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
AERCap: First Error Pointer: 00, ECRCGenCap- ECRCGenEn- ECRCChkCap- ECRCChkEn-
MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
HeaderLog: 04000001 0000000f 20010000 4a412001
Capabilities: [1e0 v1] Secondary PCI Express
LnkCtl3: LnkEquIntrruptEn- PerformEqu-
LaneErrStat: 0
Capabilities: [1c0 v1] Power Budgeting <?>
Capabilities: [148 v1] Alternative Routing-ID Interpretation (ARI)
ARICap: MFVC- ACS-, Next Function: 0
ARICtl: MFVC- ACS-, Function Group: 0
Kernel modules: megaraid_sasWhen compared, one of the notable changes is that proxmox is missing the line: Kernel driver in use: megaraid_sas
But the easy approach does not work:
```
Code:
root@homeserver:~# lsmod | grep megaraid_sas
megaraid_sas 192512 0
root@homeserver:~# modprobe megaraid_sas
root@homeserver:~# lsmod | grep megaraid_sas
megaraid_sas 192512 0```
The dmsg looks like this
Code:
dmesg | grep -i megaraid
[ 3.069159] megaraid_sas 0000:01:00.0: BAR:0x1 BAR's base_addr(phys):0x00000000dfb00000 mapped virt_addr:0x0000000012019caf
[ 3.069180] megaraid_sas 0000:01:00.0: FW now in Ready state
[ 3.069189] megaraid_sas 0000:01:00.0: 63 bit DMA mask and 32 bit consistent mask
[ 3.069928] megaraid_sas 0000:01:00.0: firmware supports msix : (96)
[ 3.073529] megaraid_sas 0000:01:00.0: requested/available msix 9/9 poll_queue 0
[ 3.073553] megaraid_sas 0000:01:00.0: current msix/online cpus : (9/8)
[ 3.073564] megaraid_sas 0000:01:00.0: RDPQ mode : (disabled)
[ 3.073575] megaraid_sas 0000:01:00.0: Current firmware supports maximum commands: 272 LDIO threshold: 237
[ 3.076907] megaraid_sas 0000:01:00.0: Performance mode :Latency (latency index = 1)
[ 3.076925] megaraid_sas 0000:01:00.0: FW supports sync cache : Yes
[ 3.076939] megaraid_sas 0000:01:00.0: megasas_disable_intr_fusion is called outbound_intr_mask:0x40000009
[ 3.274539] megaraid_sas 0000:01:00.0: FW provided supportMaxExtLDs: 0 max_lds: 32
[ 3.274553] megaraid_sas 0000:01:00.0: controller type : iMR(0MB)
[ 3.274561] megaraid_sas 0000:01:00.0: Online Controller Reset(OCR) : Enabled
[ 3.274566] megaraid_sas 0000:01:00.0: Secure JBOD support : Yes
[ 3.274571] megaraid_sas 0000:01:00.0: NVMe passthru support : No
[ 3.274576] megaraid_sas 0000:01:00.0: FW provided TM TaskAbort/Reset timeout : 0 secs/0 secs
[ 3.274582] megaraid_sas 0000:01:00.0: JBOD sequence map support : Yes
[ 3.274586] megaraid_sas 0000:01:00.0: PCI Lane Margining support : No
[ 8.804519] megaraid_sas 0000:01:00.0: megasas_get_ld_map_info DCMD timed out, RAID map is disabled
[ 8.804582] megaraid_sas 0000:01:00.0: megasas_enable_intr_fusion is called outbound_intr_mask:0x40000000
[ 8.804583] megaraid_sas 0000:01:00.0: INIT adapter done
[ 189.485559] megaraid_sas 0000:01:00.0: DCMD(opcode: 0x200e102) is timed out, func:megasas_issue_blocked_cmd
[ 189.485575] megaraid_sas 0000:01:00.0: megasas_sync_pd_seq_num DCMD timed out, continue without JBOD sequence map
[ 369.709609] megaraid_sas 0000:01:00.0: DCMD(opcode: 0x2010100) is timed out, func:megasas_issue_blocked_cmd
[ 369.709613] megaraid_sas 0000:01:00.0: Ignore DCMD timeout: megasas_get_pd_list 4694
[ 549.565521] megaraid_sas 0000:01:00.0: DCMD(opcode: 0x3010100) is timed out, func:megasas_issue_blocked_cmd
[ 549.565536] megaraid_sas 0000:01:00.0: Ignore DCMD timeout: megasas_ld_list_query 4940
[ 549.565545] megaraid_sas 0000:01:00.0: failed to get LD list
[ 549.565549] megaraid_sas 0000:01:00.0: megasas_init_fw: megasas_get_device_list failed
[ 549.565557] megaraid_sas 0000:01:00.0: megasas_disable_intr_fusion is called outbound_intr_mask:0x40000009
[ 549.566212] megaraid_sas 0000:01:00.0: Failed from megasas_init_fw 6539I reinstalled the operating systems about 6 times and every time the same problem occurs. any ideas how I could make these disks available under proxmox.