Intermittent SAS device blocking on three HBA330 hosts — mpt3sas 54.100.00.00, kernel 7.0.14-16-pve, Ceph slow ops

[Kiemal]

Hello,

I am investigating intermittent blocking of multiple local SAS SSDs on a three-node Proxmox VE/Ceph cluster. Similar incidents have now occurred on all three hosts.

I would appreciate advice on whether this resembles a known mpt3sas issue and what additional diagnostics would help distinguish a driver/firmware interaction from a hardware problem.

Environment:

• Proxmox VE GUI reports version 9.2.18.
• Kernel on all three nodes: 7.0.14-16-pve.
• mpt3sas module version on all nodes: 54.100.00.00.
• Dell HBA330 / Broadcom-LSI SAS3008.
• Core HBA firmware reported by Linux on all nodes: 16.00.11.00.
• HBA BIOS version reported on pve3: 18.00.03.00.
• Ceph 20.2.4, 24 SSD OSDs across three nodes, eight OSDs per node.
• BlueStore DB/WAL are on the OSD SSDs, without separate DB/WAL devices.

Observed incidents (all timestamps CET, UTC+1):

• pve1: 2026-10-29, approximately 07:47:08–07:47:36; six devices blocked and subsequently unblocked.
• pve2: 2026-11-03, approximately 19:49:02–19:49:40; four devices blocked and subsequently unblocked.
• pve3: 2026-11-03, approximately 22:02:59–22:03:58; seven devices blocked and subsequently unblocked.

Individual device blocks last approximately 13–19 seconds. On pve2 and pve3, the complete kernel excerpts show repeated TEST_UNIT_READY and START_UNIT messages between device_block and device_unblocked.

Example from pve3:

Nov 03 22:02:59 pve3 kernel: sd 0:0:0:0: device_block, handle(0x000a)
Nov 03 22:03:01 pve3 kernel: TEST_UNIT_READY: handle(0x000a) lun(0)
Nov 03 22:03:02 pve3 kernel: START_UNIT: handle(0x000a), lun(0)
[repeated TEST_UNIT_READY / START_UNIT messages]
Nov 03 22:03:13 pve3 kernel: sd 0:0:0:0: device_unblocked, handle(0x000a)
Other devices enter the same sequence during the incident. The inspected kernel excerpts do not show an explicit controller-wide hard reset or IOC fault.

Ceph impact:

• On pve3, osd.17, osd.18 and osd.22 reported BLUESTORE_SLOW_OP_ALERT.
• BlueStore kv_commit/kv_sync/_txc_committed_kv latencies were approximately 8.94–10.45 seconds, coinciding with the SAS blocking.
• During the earlier pve1 incident, BlueStore reported approximately 16–20-second delays.
• After the pve3 incident, all 24 OSDs remained up/in and all 1025 PGs were active+clean.
• ceph osd perf subsequently showed 0 ms at near-idle load.
• pvestatd reported status update durations of 16.775 and 20.929 seconds during the pve3 incident.

Diagnostics completed on pve3:

1. Eight SAS SSDs:
   
   • Four HGST HUSMR1616ASS200, firmware A302.
   • Four HGST HUSMM1616ASS205, firmware K326.
   • All SAS links negotiated at 12 Gbit/s.
2. SMART:
   
   • All eight devices report SMART Health Status: OK.
   • No uncorrected read/write/verify errors.
   • Grown defect list: 0 on every drive.
   • Non-medium error count: 0 on every drive.
   • Reported endurance used: 0–4%.
   • Temperatures: 21–24°C.
3. SAS PHY counters:
   
   • Small non-zero Invalid DWORD, Running Disparity and Loss of DWORD Synchronization counters exist.
   • These counters were identical in two readings approximately 35 minutes apart.
   • We cannot establish when the historical counter increments occurred.
   • Interestingly, the drive with the highest synchronization-loss count was not among the seven devices blocked in this particular pve3 incident.
4. No corresponding event was found in the pve3 iDRAC Lifecycle Log.
5. No matching systemd timer execution or obvious triggering service event was found around the pve3 incident.
6. The service-name check found smartmontools running, but no matching Dell OMSA/iSM/SupportAssist services. smartd configuration is:
   
   DEVICESCAN -d removable -n standby -m root -M exec /usr/share/smartmontools/smartd-runner
   
   There is no scheduled self-test option in this configuration and no smartmontools journal entry in the inspected incident window. This does not rule out routine polling.

There was no manual maintenance on the Proxmox cluster at the time of the pve3 incident. We have not changed driver parameters, disabled SMART monitoring, flashed firmware or rebooted nodes as a workaround.

Questions:

• Is this blocking/readiness-retry pattern known with mpt3sas 54.100.00.00 and kernel 7.0.14-16-pve on SAS3008/HBA330?
• What read-only diagnostics or targeted tracing would identify the firmware event or SCSI condition that initially triggers device_block?
• Is there a relevant supported kernel fix or recommended diagnostic comparison?
• Which evidence would help distinguish HBA/backplane/link problems from a driver or firmware interaction?

Full kernel incident excerpts and SMART/SAS PHY reports are available and can be attached.

Thank you.

 

[EllerholdAG]

Hello,

Does this happen with the newest enterprise kernel too?