mpt3sas breaks deep sleep

T

Tail870

Member
Feb 28, 2021
6
0
21
Hello!
I'm using Proxmox VE 8.3-1 with LSI 9200-i8 card (this exact model on AliExpress).
When I'm trying to wake up host from suspend mode (systemctl suspend) it just resets. Tried kernels 6.8 (default one) and 6.11.
At this moment only thing helps is blacklisting mpt3sas module.
But sleep mode works perfectly on Debian 12.10 (kernel 6.1), Ubunttu 24.10 (kernel 6.11), WIndows 11.

On all Linux-based OSese slep mode is deep:
Bash: 
:~# cat /sys/power/mem_sleep
s2idle [deep]

My hardware:
  • CPU: AMD Ryzen 5600G
  • Mobo: Aorus B550i Pro AX
  • RAM: 64 GB
    NVMe: kingston kc3000 512 GB
  • PCI-E: LSI 9200-i8 with some random HDDs for testing
DMESG output for Proxmox (filtered):
Bash: 
:~# dmesg | grep -i -e mpt2 -e mpt3 -e sas -e lsi
[    3.526039] mpt3sas version 48.100.00.00 loaded
[    3.526292] mpt2sas_cm0: 64 BIT PCI BUS DMA ADDRESSING SUPPORTED, total mem (63644364 kB)
[    3.576243] mpt2sas_cm0: CurrentHostPageSize is 0: Setting default host page size to 4k
[    3.576251] mpt2sas_cm0: MSI-X vectors supported: 1
[    3.576254] mpt2sas_cm0:  0 1 1
[    3.576331] mpt2sas_cm0: High IOPs queues : disabled
[    3.576333] mpt2sas0-msix0: PCI-MSI-X enabled: IRQ 40
[    3.576334] mpt2sas_cm0: iomem(0x00000000c04c0000), mapped(0x000000006466272a), size(16384)
[    3.576337] mpt2sas_cm0: ioport(0x000000000000f000), size(256)
[    3.629283] mpt2sas_cm0: CurrentHostPageSize is 0: Setting default host page size to 4k
[    3.658047] mpt2sas_cm0: scatter gather: sge_in_main_msg(1), sge_per_chain(9), sge_per_io(128), chains_per_io(15)
[    3.658111] mpt2sas_cm0: request pool(0x00000000062e8b7a) - dma(0xfff80000): depth(3492), frame_size(128), pool_size(436 kB)
[    3.663214] mpt2sas_cm0: sense pool(0x00000000f70c955d) - dma(0xff900000): depth(3367), element_size(96), pool_size (315 kB)
[    3.663247] mpt2sas_cm0: reply pool(0x00000000d0aadbba) - dma(0xff880000): depth(3556), frame_size(128), pool_size(444 kB)
[    3.663255] mpt2sas_cm0: config page(0x0000000000dc91f6) - dma(0xff87b000): size(512)
[    3.663256] mpt2sas_cm0: Allocated physical memory: size(7579 kB)
[    3.663258] mpt2sas_cm0: Current Controller Queue Depth(3364),Max Controller Queue Depth(3432)
[    3.663259] mpt2sas_cm0: Scatter Gather Elements per IO(128)
[    3.714045] mpt2sas_cm0: LSISAS2008: FWVersion(20.00.07.00), ChipRevision(0x03)
[    3.714051] mpt2sas_cm0: Protocol=(Initiator,Target), Capabilities=(TLR,EEDP,Snapshot Buffer,Diag Trace Buffer,Task Set Full,NCQ)
[    3.714139] scsi host6: Fusion MPT SAS Host
[    3.714761] mpt2sas_cm0: sending port enable !!
[    5.216224] mpt2sas_cm0: hba_port entry: 00000000a69cc86e, port: 255 is added to hba_port list
[    5.218160] mpt2sas_cm0: host_add: handle(0x0001), sas_addr(0x500605b005493b00), phys(8)
[    5.218832] mpt2sas_cm0: handle(0x9) sas_address(0x4433221100000000) port_type(0x1)
[    5.219320] mpt2sas_cm0: handle(0xa) sas_address(0x4433221103000000) port_type(0x1)
[   11.347470] mpt2sas_cm0: port enable: SUCCESS
[   11.550375] scsi 6:0:0:0: SATA: handle(0x0009), sas_addr(0x4433221100000000), phy(0), device_name(0x0000000000000000)
[   12.518018]  end_device-6:0: add: handle(0x0009), sas_addr(0x4433221100000000)
[   12.716854] scsi 6:0:1:0: SATA: handle(0x000a), sas_addr(0x4433221103000000), phy(3), device_name(0x0000000000000000)
[   12.735856]  end_device-6:1: add: handle(0x000a), sas_addr(0x4433221103000000)

DMESG output for Ubuntu (filtered):
Bash: 
:~$ sudo dmesg | grep -i -e mpt2 -e mpt3 -e sas -e lsi
[    3.636955] mpt3sas version 48.100.00.00 loaded
[    3.638703] mpt2sas_cm0: 64 BIT PCI BUS DMA ADDRESSING SUPPORTED, total mem (63640860 kB)
[    3.682507] mpt2sas_cm0: CurrentHostPageSize is 0: Setting default host page size to 4k
[    3.682783] mpt2sas_cm0: MSI-X vectors supported: 1
[    3.683358] mpt2sas_cm0:  0 1 1
[    3.683715] mpt2sas_cm0: High IOPs queues : disabled
[    3.684150] mpt2sas0-msix0: PCI-MSI-X enabled: IRQ 66
[    3.684686] mpt2sas_cm0: iomem(0x00000000c04c0000), mapped(0x0000000094cff283), size(16384)
[    3.685220] mpt2sas_cm0: ioport(0x000000000000f000), size(256)
[    3.738526] mpt2sas_cm0: CurrentHostPageSize is 0: Setting default host page size to 4k
[    3.766013] mpt2sas_cm0: scatter gather: sge_in_main_msg(1), sge_per_chain(9), sge_per_io(128), chains_per_io(15)
[    3.766422] mpt2sas_cm0: request pool(0x00000000da81119a) - dma(0xfff80000): depth(3492), frame_size(128), pool_size(436 kB)
[    3.772742] mpt2sas_cm0: sense pool(0x00000000a4074114) - dma(0xff900000): depth(3367), element_size(96), pool_size (315 kB)
[    3.773077] mpt2sas_cm0: reply pool(0x00000000bfefbb47) - dma(0xff880000): depth(3556), frame_size(128), pool_size(444 kB)
[    3.773301] mpt2sas_cm0: config page(0x00000000d83c8fe5) - dma(0xff87b000): size(512)
[    3.773510] mpt2sas_cm0: Allocated physical memory: size(7579 kB)
[    3.773715] mpt2sas_cm0: Current Controller Queue Depth(3364),Max Controller Queue Depth(3432)
[    3.773922] mpt2sas_cm0: Scatter Gather Elements per IO(128)
[    3.823049] mpt2sas_cm0: LSISAS2008: FWVersion(20.00.07.00), ChipRevision(0x03)
[    3.823448] mpt2sas_cm0: Protocol=(Initiator,Target), Capabilities=(TLR,EEDP,Snapshot Buffer,Diag Trace Buffer,Task Set Full,NCQ)
[    3.823985] scsi host0: Fusion MPT SAS Host
[    3.825103] mpt2sas_cm0: sending port enable !!
[    5.398262] mpt2sas_cm0: hba_port entry: 000000002e6dfe02, port: 255 is added to hba_port list
[    5.400480] mpt2sas_cm0: host_add: handle(0x0001), sas_addr(0x500605b005493b00), phys(8)
[    5.401473] mpt2sas_cm0: handle(0x9) sas_address(0x4433221100000000) port_type(0x1)
[    5.402261] mpt2sas_cm0: handle(0xa) sas_address(0x4433221103000000) port_type(0x1)
[   11.528816] mpt2sas_cm0: port enable: SUCCESS
[   11.734447] scsi 0:0:0:0: SATA: handle(0x0009), sas_addr(0x4433221100000000), phy(0), device_name(0x0000000000000000)
[   12.701842]  end_device-0:0: add: handle(0x0009), sas_addr(0x4433221100000000)
[   12.899800] scsi 0:0:1:0: SATA: handle(0x000a), sas_addr(0x4433221103000000), phy(3), device_name(0x0000000000000000)
[   12.917384]  end_device-0:1: add: handle(0x000a), sas_addr(0x4433221103000000)
 
Last edited:
Info about PCI-device:
Code: 
:~# lspci -vvv -s 01:00.0
01:00.0 Serial Attached SCSI controller: Broadcom / LSI SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon] (rev 03)
        Subsystem: IBM SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon]
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0, Cache Line Size: 64 bytes
        Interrupt: pin A routed to IRQ 38
        IOMMU group: 9
        Region 0: I/O ports at f000 [size=256]
        Region 1: Memory at c04c0000 (64-bit, non-prefetchable) [size=16K]
        Region 3: Memory at c0080000 (64-bit, non-prefetchable) [size=256K]
        Expansion ROM at c0000000 [disabled] [size=512K]
        Capabilities: [50] Power Management version 3
                Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
                Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
        Capabilities: [68] Express (v2) Endpoint, MSI 00
                DevCap: MaxPayload 4096 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us
                        ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0W
                DevCtl: CorrErr+ NonFatalErr+ FatalErr+ UnsupReq+
                        RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+ FLReset-
                        MaxPayload 512 bytes, MaxReadReq 512 bytes
                DevSta: CorrErr+ NonFatalErr- FatalErr- UnsupReq+ AuxPwr- TransPend-
                LnkCap: Port #0, Speed 5GT/s, Width x8, ASPM L0s, Exit Latency L0s <64ns
                        ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp-
                LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed 5GT/s, Width x8
                        TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
                DevCap2: Completion Timeout: Range BC, TimeoutDis+ NROPrPrP- LTR-
                         10BitTagComp- 10BitTagReq- OBFF Not Supported, ExtFmt- EETLPPrefix-
                         EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
                         FRS- TPHComp- ExtTPHComp-
                         AtomicOpsCap: 32bit- 64bit- 128bitCAS-
                DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR- 10BitTagReq- OBFF Disabled,
                         AtomicOpsCtl: ReqEn-
                LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
                         Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
                         Compliance Preset/De-emphasis: -6dB de-emphasis, 0dB preshoot
                LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete- EqualizationPhase1-
                         EqualizationPhase2- EqualizationPhase3- LinkEqualizationRequest-
                         Retimer- 2Retimers- CrosslinkRes: unsupported
        Capabilities: [d0] Vital Product Data
pcilib: sysfs_read_vpd: read failed: No such device
                Not readable
        Capabilities: [a8] MSI: Enable- Count=1/1 Maskable- 64bit+
                Address: 0000000000000000  Data: 0000
        Capabilities: [c0] MSI-X: Enable+ Count=15 Masked-
                Vector table: BAR=1 offset=00002000
                PBA: BAR=1 offset=00003800
        Capabilities: [100 v1] Advanced Error Reporting
                UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
                CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr-
                CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
                AERCap: First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn-
                        MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
                HeaderLog: 04000001 0000220f 01070000 1235be54
        Capabilities: [138 v1] Power Budgeting <?>
        Capabilities: [150 v1] Single Root I/O Virtualization (SR-IOV)
                IOVCap: Migration- 10BitTagReq- Interrupt Message Number: 000
                IOVCtl: Enable- Migration- Interrupt- MSE- ARIHierarchy+ 10BitTagReq-
                IOVSta: Migration-
                Initial VFs: 16, Total VFs: 16, Number of VFs: 0, Function Dependency Link: 00
                VF offset: 1, stride: 1, Device ID: 0072
                Supported Page Size: 00000553, System Page Size: 00000001
                Region 0: Memory at 00000000c04c4000 (64-bit, non-prefetchable)
                Region 2: Memory at 00000000c00c0000 (64-bit, non-prefetchable)
                VF Migration: offset: 00000000, BIR: 0
        Capabilities: [190 v1] Alternative Routing-ID Interpretation (ARI)
                ARICap: MFVC- ACS-, Next Function: 0
                ARICtl: MFVC- ACS-, Function Group: 0
        Kernel driver in use: mpt3sas
        Kernel modules: mpt3sas

:~#
 
I just tested installing Proxmox on Debian 12 w/o removing kernel (so I can test it with Proxmox packages). Both Debian 6.1 and Proxmox 6.8 kernels now resets host instead of waking from suspend... Which means that kernel itself shouldn't be the culprit.
 
Last edited:
Both Debian 6.1 and Proxmox 6.8 kernels
Click to expand...
It just hit me that mpt3sas not the main reason why suspend is broken since that issue occured on both kernels but something in Proxmox in general...
 
Last edited:
You must log in or register to reply here.
Top Bottom