New FC script is now available.
The latest updates focus not only on monitoring connectivity and path status, but also on identifying potential I/O problems and correlating events from different Linux subsystems.
1. SCSI device state monitoring
The script now monitors the actual SCSI device state through /sys/block/sdX/device/state.
It identifies states such as:
- running
- blocked
- offline
- transport-offline
- quiesce
- Other transitional device states
Non-running devices are reported separately in a
SCSI ALERT section, including the SCSI device and associated HBA host.
This is particularly useful because a path can appear active in Multipath while the corresponding SCSI device is actually blocked.
2. I/O performance and troubleshooting metrics
Additional I/O statistics are displayed for individual SCSI devices:
- IOPS – completed I/O operations per second.
- In Progress – number of I/O requests currently being processed.
- IO Utilization – percentage of the measurement interval during which the device had outstanding I/O requests.
These metrics help identify potential I/O stalls.
For example, the following combination is particularly interesting:
- IOPS = 0
- In Progress = 20
- IO Utilization = 100%
- SCSI state = blocked
This indicates that I/O requests remain outstanding without completed operations during the measurement interval. It can be an important indicator of a possible storage path or I/O processing problem.
3. Improved FC and SCSI event monitoring
The event monitoring section collects information from two sources:
- Linux kernel journal (-k)
- multipathd journal
Events are classified into three levels:
- CRITICAL – serious transport and I/O errors, such as DID_TRANSPORT_DISRUPTED, DID_NO_CONNECT, DID_TIME_OUT, I/O error and all paths down.
- WARNING – path failures, command aborts, device resets, checker failures and FC link-down events.
- RECOVERY – path reinstatement, link-up events and successful path recovery.
4. Event grouping and duplicate reduction
Related events for the same SCSI device and HBA host are grouped within a 120-second time window.
For example, events such as:
- path down
- mark as failed
- DID_TRANSPORT_DISRUPTED
- path up
- reinstated
can be displayed together rather than as multiple independent log entries.
Duplicate event messages within a grouped incident are also filtered.
The display retains the latest 12 grouped events.
5. Improved SCSI ALERT section
The script separately reports SCSI devices whose current state is not running.
For example:
========================================================================
SCSI ALERT: sdd (host1) BLOCKED
========================================================================
This provides an immediate indication of a problematic SCSI path, independently of the Multipath path checker status.
6. Improved Multipath information
The Multipath output has been improved to make device identification clearer.
The device information now distinguishes the
Vendor/Product field from the Multipath alias or WWID, avoiding confusion between the storage device identification and the logical multipath device name.
7. FC traffic monitoring
The script also monitors FC port traffic:
- TX – transmitted traffic.
- RX – received traffic.
Values are calculated using the FC port statistics (tx_words and rx_words), with each FC word representing four bytes.
This provides a useful overview of traffic per FC HBA port.
Why these improvements matter
One of the main goals was to correlate information from different Linux layers instead of relying only on the Multipath status.
For example, it is possible to have:
- FC HBA port: Online
- FC remote port: Online
- ALUA: active/optimized
- Multipath path: active
- SCSI device: blocked
These states are not necessarily contradictory. They describe different layers of the storage stack.
By combining SCSI state, I/O statistics, FC traffic and kernel/multipathd events, fc-monitor.sh provides more information for investigating intermittent FC SAN connectivity and I/O problems.
The script is still intended as a monitoring and troubleshooting aid, not as a replacement for the storage vendor's diagnostic tools or detailed FC switch and SAN analysis.
Script is available at:
https://github.com/hhorvat/scripts/blob/main/fc-monitor.sh
