Intermittent SAS device blocking on three HBA330 hosts — mpt3sas 54.100.00.00, kernel 7.0.14-16-pve, Ceph slow ops

K

Kiemal

New Member
Nov 4, 2026
3
0
1
Hello,

I am investigating intermittent blocking of multiple local SAS SSDs on a three-node Proxmox VE/Ceph cluster. Similar incidents have now occurred on all three hosts.

I would appreciate advice on whether this resembles a known mpt3sas issue and what additional diagnostics would help distinguish a driver/firmware interaction from a hardware problem.

Environment:

  • Proxmox VE GUI reports version 9.2.18.
  • Kernel on all three nodes: 7.0.14-16-pve.
  • mpt3sas module version on all nodes: 54.100.00.00.
  • Dell HBA330 / Broadcom-LSI SAS3008.
  • Core HBA firmware reported by Linux on all nodes: 16.00.11.00.
  • HBA BIOS version reported on pve3: 18.00.03.00.
  • Ceph 20.2.4, 24 SSD OSDs across three nodes, eight OSDs per node.
  • BlueStore DB/WAL are on the OSD SSDs, without separate DB/WAL devices.
Observed incidents (all timestamps CET, UTC+1):

  • pve1: 2026-10-29, approximately 07:47:08–07:47:36; six devices blocked and subsequently unblocked.
  • pve2: 2026-11-03, approximately 19:49:02–19:49:40; four devices blocked and subsequently unblocked.
  • pve3: 2026-11-03, approximately 22:02:59–22:03:58; seven devices blocked and subsequently unblocked.
Individual device blocks last approximately 13–19 seconds. On pve2 and pve3, the complete kernel excerpts show repeated TEST_UNIT_READY and START_UNIT messages between device_block and device_unblocked.

Example from pve3:

Nov 03 22:02:59 pve3 kernel: sd 0:0:0:0: device_block, handle(0x000a)
Nov 03 22:03:01 pve3 kernel: TEST_UNIT_READY: handle(0x000a) lun(0)
Nov 03 22:03:02 pve3 kernel: START_UNIT: handle(0x000a), lun(0)
[repeated TEST_UNIT_READY / START_UNIT messages]
Nov 03 22:03:13 pve3 kernel: sd 0:0:0:0: device_unblocked, handle(0x000a)
Other devices enter the same sequence during the incident. The inspected kernel excerpts do not show an explicit controller-wide hard reset or IOC fault.

Ceph impact:

  • On pve3, osd.17, osd.18 and osd.22 reported BLUESTORE_SLOW_OP_ALERT.
  • BlueStore kv_commit/kv_sync/_txc_committed_kv latencies were approximately 8.94–10.45 seconds, coinciding with the SAS blocking.
  • During the earlier pve1 incident, BlueStore reported approximately 16–20-second delays.
  • After the pve3 incident, all 24 OSDs remained up/in and all 1025 PGs were active+clean.
  • ceph osd perf subsequently showed 0 ms at near-idle load.
  • pvestatd reported status update durations of 16.775 and 20.929 seconds during the pve3 incident.
Diagnostics completed on pve3:

  1. Eight SAS SSDs:
    • Four HGST HUSMR1616ASS200, firmware A302.
    • Four HGST HUSMM1616ASS205, firmware K326.
    • All SAS links negotiated at 12 Gbit/s.
  2. SMART:
    • All eight devices report SMART Health Status: OK.
    • No uncorrected read/write/verify errors.
    • Grown defect list: 0 on every drive.
    • Non-medium error count: 0 on every drive.
    • Reported endurance used: 0–4%.
    • Temperatures: 21–24°C.
  3. SAS PHY counters:
    • Small non-zero Invalid DWORD, Running Disparity and Loss of DWORD Synchronization counters exist.
    • These counters were identical in two readings approximately 35 minutes apart.
    • We cannot establish when the historical counter increments occurred.
    • Interestingly, the drive with the highest synchronization-loss count was not among the seven devices blocked in this particular pve3 incident.
  4. No corresponding event was found in the pve3 iDRAC Lifecycle Log.
  5. No matching systemd timer execution or obvious triggering service event was found around the pve3 incident.
  6. The service-name check found smartmontools running, but no matching Dell OMSA/iSM/SupportAssist services. smartd configuration is:

    DEVICESCAN -d removable -n standby -m root -M exec /usr/share/smartmontools/smartd-runner

    There is no scheduled self-test option in this configuration and no smartmontools journal entry in the inspected incident window. This does not rule out routine polling.
There was no manual maintenance on the Proxmox cluster at the time of the pve3 incident. We have not changed driver parameters, disabled SMART monitoring, flashed firmware or rebooted nodes as a workaround.

Questions:

  • Is this blocking/readiness-retry pattern known with mpt3sas 54.100.00.00 and kernel 7.0.14-16-pve on SAS3008/HBA330?
  • What read-only diagnostics or targeted tracing would identify the firmware event or SCSI condition that initially triggers device_block?
  • Is there a relevant supported kernel fix or recommended diagnostic comparison?
  • Which evidence would help distinguish HBA/backplane/link problems from a driver or firmware interaction?
Full kernel incident excerpts and SMART/SAS PHY reports are available and can be attached.

Thank you.
 

Attachments

Hello, thanks for your reply.

All three nodes are currently running kernel 7.0.14-16-pve. I have not tested a different kernel yet.

I do not have a Proxmox subscription or access to the enterprise repository. Could you please specify the exact kernel version you would like me to test, and whether that version is also available through the no-subscription repository?

This is a production three-node Ceph cluster, so I would need to schedule any kernel change and reboot one node at a time.
 
Thank you for pointing me to the newer kernel. I confirmed that 7.0.14-26-pve is available in the no-subscription repository and have now installed it on pve1. The other two nodes remain on 7.0.14-16-pve for comparison.

There was an unexpected issue after the reboot: all eight OSDs on pve1 were inactive, although all SAS disks and their LVM volumes were visible. The journal showed the OSD services starting and then immediately reporting “Deactivated successfully”. For osd.0, systemd reported:

ExecStart=/usr/bin/ceph-osd -f --id 0 --setuser ceph --setgroup ceph (code=killed, signal=HUP)

Starting osd.0 manually worked. I then manually started the remaining seven OSDs, and all of them rejoined the cluster. The cluster is now back to HEALTH_OK, with 24/24 OSDs up/in and all 1025 PGs active+clean. The temporary host-level noout flag has been removed.

I have not yet identified the source of the HUP signal, so I cannot attribute this startup issue to the new kernel.

I will now observe pve1 for recurrence of the original intermittent SAS stalls / BLUESTORE_SLOW_OP_ALERT. It is too early to say whether the kernel change resolves them.

Has anyone seen this OSD startup behaviour before, or could you suggest what to check next?

Thanks again for your help.
 
Since Dell HBA 330s are for 13th & 14th-gen Dells, the latest firmware version is 16.17.01.00. I'm running these in production without issue. I've also configured the controller to boot in UEFI mode only. Another thing to check is that X2APIC, I/OAT DMA is enabled as well in the BIOS. I don't have SRV-IO enabled because I don't need it. I am running the latest PVE kernel of 7.0.14-28 in production.
 
You must log in or register to reply here.
Top Bottom
Back