Proxmox VE 9.2 – sporadic iSCSI NOP timeouts / DID_TRANSPORT_DISRUPTED with Dell PowerVault ME5 and Intel E810

[huganega]

Hi,

I am experiencing sporadic iSCSI connection interruptions on a Proxmox VE 9.2 host connected directly to a Dell PowerVault ME5224.
The issue manifests as ISCSI_ERR_NOP_TIMEDOUT, followed by DID_TRANSPORT_DISRUPTED. The affected iSCSI session usually recovers automatically after a few seconds.
The problem is intermittent. The system can run normally for a considerable amount of time before another interruption occurs.

Environment

• Proxmox VE 9.2
• currently running kernel: 7.0.14-11-pve
• 7.0.14-12-pve already installed, but not yet tested
• Dell PowerVault ME5224, dual controller
• ME5 firmware: ME5.2.0.1.1
• 25 GbE iSCSI
• direct attached iSCSI connections, no Ethernet switch between host and storage
• Intel E810-XXV 25 GbE adapters (ice driver)
• MTU 9000
• two independent iSCSI sessions / paths
• device-mapper multipath with ALUA
• LVM on top of the multipath device

The LUN WWID is:
3600c0ff0005d88902d02456a01000000

Current multipath state:
3600c0ff0005d88902d02456a01000000 dm-0 DellEMC,ME5
size=23T features='0' hwhandler='1 alua' wp=rw
|-+- policy='service-time 0' prio=50 status=active
| `- 3:0:0:0 sdh 8:112 active ready running
`-+- policy='service-time 0' prio=10 status=enabled
`- 2:0:0:0 sdg 8:96 active ready running

Multipath configuration

I have reverted the ME5 device configuration to the settings recommended in Dell's Linux documentation:

defaults {
polling_interval 5
checker_timeout 15
find_multipaths no
user_friendly_names no
enable_foreign nvme
}

devices {
device {
vendor "DellEMC"
product "ME5"
path_grouping_policy "group_by_prio"
path_checker "tur"
hardware_handler "1 alua"
prio "alua"
failback immediate
rr_weight "uniform"
path_selector "service-time 0"
}
}

Local SAS disks are explicitly blacklisted. Only the ME5 LUN appears in multipath -ll.

iSCSI settings

node.session.timeo.replacement_timeout = 5
node.conn[0].timeo.noop_out_interval = 2
node.conn[0].timeo.noop_out_timeout = 10
node.session.cmds_max = 1024
node.session.queue_depth = 128
node.session.iscsi.InitialR2T = Yes
node.session.iscsi.ImmediateData = No

Both SCSI paths currently have:
queue_depth = 128

and the multipath device:
nr_requests = 256
max_sectors_kb = 4096

Example of an actual failure
Aug 19 15:44:08 pve1 kernel: connection6:0: ping timeout of 10 secs expired, recv timeout 2, last rx 4940010531, last ping 4940010361, now 4940022785
Aug 19 15:44:08 pve1 kernel: connection6:0: detected conn error (1022)
Aug 19 15:44:09 pve1 iscsid[3208]: Kernel reported iSCSI connection 6:0 error (1022 - ISCSI_ERR_NOP_TIMEDOUT: A NOP has timed out) state (3)
Aug 19 15:44:12 pve1 iscsid[3208]: connection6:0 is operational after recovery (1 attempts)
Aug 19 15:44:26 pve1 kernel: connection6:0: ping timeout of 10 secs expired, recv timeout 2, last rx 4940028114, last ping 4940027920, now 4940040193
Aug 19 15:44:26 pve1 kernel: connection6:0: detected conn error (1022)
Aug 19 15:44:26 pve1 kernel: sd 3:0:0:0: [sdh] tag#182 FAILED Result: hostbyte=DID_TRANSPORT_DISRUPTED driverbyte=DRIVER_OK cmd_age=14s
Aug 19 15:44:26 pve1 kernel: sd 3:0:0:0: [sdh] tag#164 FAILED Result: hostbyte=DID_TRANSPORT_DISRUPTED driverbyte=DRIVER_OK cmd_age=14s
Aug 19 15:44:26 pve1 kernel: sd 3:0:0:0: [sdh] tag#182 CDB: Test Unit Ready 00 00 00 00 00 00
Aug 19 15:44:26 pve1 kernel: sd 3:0:0:0: [sdh] tag#164 CDB: Test Unit Ready 00 00 00 00 00 00
Aug 19 15:44:26 pve1 iscsid[3208]: Kernel reported iSCSI connection 6:0 error (1022 - ISCSI_ERR_NOP_TIMEDOUT: A NOP has timed out) state (3)
Aug 19 15:44:29 pve1 iscsid[3208]: connection6:0 is operational after recovery (1 attempts)
Aug 19 15:44:31 pve1 kernel: sd 3:0:0:0: alua: supports implicit TPGS
Aug 19 15:44:31 pve1 kernel: sd 3:0:0:0: alua: device naa.600c0ff0005d88902d02456a01000000 port group 0 rel port 1


During this event pvestatd was also delayed:


Aug 19 15:44:31 pve1 pvestatd[3611]: status update time (27.868 seconds)


NIC / physical link

I checked the Intel E810 interface counters after the event. There are no CRC errors, packet drops or TX timeouts:
rx_dropped: 0
tx_errors: 0
tx_timeout.nic: 0
rx_dropped.nic: 0
rx_crc_errors.nic: 0


The kernel log around the failure does not contain an ice reset, link-down/link-up event or other obvious NIC error.
The affected Ethernet interface remained UP,LOWER_UP.
The second iSCSI path remains available, and multipath normally handles the failed path until the session recovers.

What has already been checked

• both iSCSI paths are present and ALUA works
• local disks are excluded from multipath
• no switch is involved in the iSCSI network
• no CRC errors or packet drops are reported by the E810
• no ice driver reset or link flap is logged at the time of the NOP timeout
• the iSCSI session automatically recovers after a few seconds
• the issue has occurred more than once and is not related to a single reboot/configuration event

I am currently trying to determine whether this is related to the Linux ice driver / E810, open-iscsi, SCSI/multipath, or the ME5 target side.
Has anyone seen similar sporadic ISCSI_ERR_NOP_TIMEDOUT / DID_TRANSPORT_DISRUPTED events with Proxmox VE 9 / kernel 7.0 and Intel E810 adapters?

Are there any known issues or recommended diagnostics for this combination?

I can provide complete iscsiadm -m session -P 3, multipathd show config, ethtool -S, kernel logs and packet captures if required.

 

[bbgeek17]

Hi @huganega ,
Are you sure that you looked at the correct network interface? The one used by iSCSI?
If you've confirmed that there are no network layer errors, then issue is likely in the higher layer of the stack. A rolling network trace would definitively tell you whether the reply is delayed by 10s or the request is never sent. The former means the issue is on the ME side (perhaps some sort of raid/group check), or PVE side (backup, CPU contention, etc).

Is this always on the same path/chain?

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[huganega]

Hi,

yes, I have now verified the path mapping again, and the sporadic NOP timeouts always affect the same iSCSI/SCSI path.
The connection number changed over time because the iSCSI sessions were recreated (connection2:0, later connection4:0, currently connection6:0), but the affected SCSI path has consistently been:

PVE nic6 / 10.10.1.2
->
ME5 10.10.1.1:3260
->
SCSI host 3
->
sdh

sdh is normally the ALUA preferred path (prio 50).


The second path:

PVE nic8 / 10.10.2.2
->
ME5 10.10.2.1:3260
->
SCSI host 2
->
sdg

remains available and is the non-preferred ALUA path (prio 10).

Examples from the kernel log:
Aug 12: connection2:0 ... DID_TRANSPORT_DISRUPTED ... sd 3:0:0:0 [sdh]
Aug 13: connection4:0 ... DID_TRANSPORT_DISRUPTED ... sd 3:0:0:0 [sdh]
Aug 14: connection6:0 ... DID_TRANSPORT_DISRUPTED ... sd 3:0:0:0 [sdh]
Aug 16: connection6:0 ... NOP timeouts
Aug 19: connection6:0 ... DID_TRANSPORT_DISRUPTED ... sd 3:0:0:0 [sdh]


I also checked the correct physical interface (nic6, Intel E810). Around the NOP event there are no ice driver resets, no link down/up events, no CRC errors, no RX drops and no TX timeouts.

Current mapping:
10.10.2.1 -> SID 5 -> host 2 -> sdg
10.10.1.1 -> SID 6 -> host 3 -> sdh

Current multipath state:
3600c0ff0005d88902d02456a01000000 dm-0 DellEMC,ME5
size=23T features='0' hwhandler='1 alua' wp=rw
|-+- policy='service-time 0' prio=50 status=active
| `- 3:0:0:0 sdh 8:112 active ready running
`-+- policy='service-time 0' prio=10 status=enabled
`- 2:0:0:0 sdg 8:96 active ready running

There was a separate failure at Aug 19 16:19 which I have identified and fixed: a PCI passthrough configuration had “All Functions” enabled and temporarily passed the second E810 port (12:00.1 / nic6) to a VM as well. I am excluding that incident from the NOP investigation.

I agree that a rolling packet capture is the logical next step. I will capture TCP/iSCSI traffic on nic6 so that, on the next occurrence, we can determine whether:

1. the NOP-Out is transmitted and the target does not reply in time,
2. the NOP-Out itself is delayed/not transmitted by the host, or
3. the reply is visible on the wire but not processed by the host in time.

Given that all unexplained events so far affect the same path, I am also investigating the physical/ME5 side of that specific path.

 

[Tmanok]

Hey Everyone,

I also checked the correct physical interface (nic6, Intel E810). Around the NOP event there are no ice driver resets, no link down/up events, no CRC errors, no RX drops and no TX timeouts.

I think there's something pretty overlooked here: Dell's ME5 documentation explicitly says that for iSCSI usage you must have symmetric flow control enabled on the interfaces. If the E810 and ME5 aren't agreeing properly on pause behavior, bursts can overrun buffers on the active path without producing:

rx_crc_errors
rx_dropped.nic
tx_errors

(on the host at least)
A target-side buffer drop obviously doesn't have to increment the host's NIC error counters. You should gather
host-port statistics from the ME5. Dell officially supports direct-connected 25 GbE iSCSI, so the topology itself isn't the problem in my opinion.

Not to let your NIC completely off the hook or anything, I just see this as less likely. If nic6 is receiving huge quantities of PAUSE frames around the incidents, that becomes interesting. Intel's own E810 documentation makes clear that the adapter has a fairly complex multi-queue/interrupt architecture, and Intel specifically recommends examining per-queue drop statistics, IRQ affinity, ring sizes, etc. when troubleshooting receive-side stalls. You should of course also gather host-port statistics from your Intel E810:

ethtool -i nic6
ethtool -i nic8
ethtool -l nic6
ethtool -g nic6
ethtool -c nic6
cat /proc/interrupts | grep -i nic6
cat /proc/net/softnet_stat

If you bring those back to the forum, we might be able to help look for firmware versions reporting issues, too. But you can always try looking through other forums or the changelogs. Here's a list of reference docs I referred to:

• https://www.dell.com/support/manual...954326-1279-4214-8440-c68f0ef2d07f&lang=en-us
• https://www.dell.com/support/manual...9e84a0-66b9-4df8-bd6b-13461a338c9b&lang=en-us
• https://www.dell.com/support/manual...69ed4a-ea80-48b8-bc30-801754ce8df3&lang=en-us
• https://www.dell.com/support/manual...c1d164-1043-4ae2-b12c-54cc8bae014f&lang=en-us
• https://github.com/intel/ethernet-linux-ice/blob/main/README (This is for Intel 800 series NICs)

Thanks, hope you find something more obvious in those port statistics!


Tmanok

 

[huganega]

Hi,

thanks for the pointers. I checked the E810 configuration and statistics.

Both iSCSI interfaces are using the ice driver with firmware 4.91 0x800214ab 1.3909.0.
Both currently use 16 combined queues, RX ring size 2048 / TX 256, and adaptive RX/TX coalescing is enabled.

Flow control also looks correct on both interfaces and, importantly, it is actually negotiated in both directions:

nic6:
RX: on
TX: on
RX negotiated: on
TX negotiated: on

nic8:
RX: on
TX: on
RX negotiated: on
TX negotiated: on

On the active path (nic6) I currently see 5,648 received XOFF and 5,648 XON frames, while there are no RX drops, CRC errors or TX timeouts:

rx_dropped: 0
tx_errors.nic: 0
tx_timeout.nic: 0
link_xon_rx.nic: 5648
link_xoff_rx.nic: 5648
rx_dropped.nic: 0
rx_crc_errors.nic: 0
mac_local_faults.nic: 0
mac_remote_faults.nic: 2

nic8 is the passive ALUA path and carries almost no I/O, so the very low pause-frame counters there seem expected.

I have also set up rolling tcpdump captures on both iSCSI interfaces. Unfortunately the NOP/ping timeout is quite sporadic, so there hasn't been another occurrence since setting this up.
For now I will leave the NIC/ring/coalescing settings unchanged and wait for the next timeout. Once it happens, I should have packet captures from both paths covering the event, together with the kernel timestamp and NIC statistics. That should allow us to determine whether the NOP-Out actually leaves the host and whether the response from the ME5 is delayed/missing.

I'll post the results as soon as I catch the next occurrence.

Thanks again for the help.

 

[huganega]

Quick update:

Unfortunately, another NOP timeout occurred last night at 03:01:30:

Aug 21 03:01:30 pve1 kernel: connection2:0: ping timeout of 10 secs expired, recv timeout 2
Aug 21 03:01:30 pve1 kernel: connection2:0: detected conn error (1022)
Aug 21 03:01:30 pve1 iscsid: Kernel reported iSCSI connection 2:0 error (1022 - ISCSI_ERR_NOP_TIMEDOUT)
Aug 21 03:01:33 pve1 iscsid: connection2:0 is operational after recovery (1 attempts)

After the reboot, SID 2 maps to:

10.10.1.2 -> 10.10.1.1

so this is again the A0/nic6 path, which is also the active ALUA path (prio 50).

The E810 statistics still don't show any obvious host-side network errors:

rx_dropped: 0
tx_errors.nic: 0
tx_timeout.nic: 0
rx_dropped.nic: 0
rx_crc_errors.nic: 0
mac_local_faults.nic: 0
mac_remote_faults.nic: 2

link_xon_rx.nic: 25127
link_xoff_rx.nic: 25127
link_xon_tx.nic: 6
link_xoff_tx.nic: 6

So flow control is clearly active on the busy path, and I still don't see packet drops, CRC errors or TX timeouts on the host NIC.

I did have rolling tcpdump captures running on both iSCSI interfaces. However, I underestimated the amount of traffic on the active 25 GbE path. By the time I noticed the event in the morning, the nic6 capture containing 03:01:30 had already been overwritten. The passive nic8 capture still contains the period, but unfortunately that is not the path on which the timeout occurred.

I have now changed the capture setup. A systemd watcher monitors the kernel log for detected conn error (1022). When another event occurs, it immediately stops both rolling captures, preserves the current PCAP rings together with iSCSI/multipath/E810 diagnostics, and then restarts the captures.

So the next occurrence should finally give us the packet trace around the actual failure and allow us to determine whether the NOP-Out leaves the PVE host and whether/when the ME5 replies.

One interesting observation so far is that all NOP timeout events I have checked have occurred on the same active A0/nic6 path. I will update the thread again as soon as the next event is captured.

 

[Tmanok]

Hi Huganega,

Ok, a few things from your output might be worth examining, but I have to ask a question I normally ignore because I'm vendor agnostic 99% of the time. Are you using a Dell recommended DAC? 25GbE has caused me trouble in the past with some products because it has additional features such as FEC. Moreover, SFP type connectors are very picky about their firmwares.

Back to your outputs and messages, there's a pretty important concept to understand here about PCAP saying that it sent NOP-Out, which is that when your ME5 is pausing transmission due to performance pressures, it may not see that NOP at all. It may genuinely be ignoring packets until it is ready again. So all that PCAP proves is that Linux sent the NOP, not that the ME5 received it.

I bring this up because your OS is clearly receiving pause requests from the ME5 (ON and OFF):

link_xon_rx.nic: 25127
link_xoff_rx.nic: 25127
link_xon_tx.nic: 6
link_xoff_tx.nic: 6

What might peg this on the ME5 for sure would be to figure out whether XOFF frames suddenly arrive in a concentrated burst immediately before the 10-second iSCSI blackout.

Something else worth considering Intel released E810 NVM 5.01 on July 10, 2026. Your firmware is fairly up to date, but it's a new NIC series and might benefit from newer firmware. Sucks that I can only find Windows utilities for it, though...
https://www.intel.com/content/www/u...net-network-adapters-e810-series-windows.html SHA256: 69DF8E77E6CDED18E38921C1626B4120A60CDAFCC69FDBAB7E90FBD8E9182C8F

tmanok@MBP14-M5 Downloads % shasum -a 256 E810_NVMUpdatePackage_v5_01_Windows.zip
69df8e77e6cded18e38921c1626b4120a60cdafcc69fdbab7e90fbd8e9182c8f  E810_NVMUpdatePackage_v5_01_Windows.zip

You can find it here since the forum does not permit large files: https://archive.org/download/e-810-...ndows/E810_NVMUpdatePackage_v5_01_Windows.zip
I linked a copy to the forum because in 6-24 months when someone tries to find this version and only finds broken links, they may thank me.

Thanks,



Tmanok

 

[huganega]

Hi Tmanok,

Thanks for pointing me towards the newer E810 NVM firmware.
I checked the adapters more closely and there is one important detail: these are HPE OEM Intel E810-XXVDA2 adapters, not Intel retail adapters.

Both adapters were running:
firmware-version: 4.91 0x800214ab 1.3909.0

HPE currently provides and recommends NVM 5.00 for these adapters. Intel already has 5.01 available for the E810 series, but since HPE has only released/validated 5.00 for the HPE OEM adapters, I decided to stay with the OEM-supported firmware rather than flashing Intel's generic 5.01 package.

I have now updated both adapters successfully to:
driver: ice
version: 7.0.14-12-pve
firmware-version: 5.00 0x80021bd4 1.4002.0

Both 25 GbE iSCSI links came back correctly after the update. Flow control is still negotiated in both directions on both interfaces:

Pause parameters for nic6:
Autonegotiate: on
RX: on
TX: on
RX negotiated: on
TX negotiated: on

Pause parameters for nic8:
Autonegotiate: on
RX: on
TX: on
RX negotiated: on
TX negotiated: on

Multipath/ALUA also looks normal, with the A0/nic6 path active at priority 50 and the B0/nic8 path available at priority 10.
After the reboot the active nic6 path is already receiving PAUSE frames again:

link_xon_rx.nic: 704
link_xoff_rx.nic: 704
rx_dropped.nic: 0
rx_crc_errors.nic: 0
tx_timeout.nic: 0

So the ME5 is clearly making use of Ethernet flow control, but there are still no obvious host-side drops or CRC errors.

One caveat: according to HPE's published fixes for NVM 5.00, the documented fix is related to LLDP receive reporting. There is no documented iSCSI, flow-control or link-stall fix that would directly explain my issue. Therefore I won't assume that the firmware update fixes the NOP timeouts.

The automated rolling PCAP/event capture is still running. I'll leave the configuration unchanged now and wait for another NOP timeout. If it happens again, we should finally have the packet trace and diagnostics from the actual event.

 

[huganega]

Hi Tmanok,
thanks again for the suggestions. I have some new data now, so I am replacing my previous update with a more complete one.

First, regarding the NIC firmware:
The adapters are HPE OEM Intel E810-XXVDA2 cards. I updated both NICs from:

firmware-version: 4.91 0x800214ab 1.3909.0

to the latest firmware currently released by HPE:

firmware-version: 5.00 0x80021bd4 1.4002.0

I deliberately did not install Intel's generic NVM 5.01 package because these are HPE OEM adapters and HPE currently only provides/recommends NVM 5.00 for them.

Both links are still running at 25 Gbit/s and flow control is negotiated RX/TX on both interfaces.

Unfortunately, the firmware update did not resolve the problem. I captured another event today:

08:43:07 connection2:0 ping timeout of 10 secs expired
08:43:07 detected conn error (1022)
08:43:07 sdk: DID_TRANSPORT_DISRUPTED
08:43:10 connection2:0 operational after recovery

08:43:24 connection2:0 ping timeout of 10 secs expired
08:43:24 detected conn error (1022)
08:43:24 sdk: DID_TRANSPORT_DISRUPTED
08:43:27 connection2:0 operational after recovery

This is again the active path via controller A0 / nic6.
The rolling PCAP capture worked as intended. For the first event tcpdump reported:

nic6: 26,519,499 packets captured
26,519,499 packets received by filter
0 packets dropped by kernel

nic8: 7,150 packets captured
7,150 packets received by filter
0 packets dropped by kernel

I also implemented a one-second logger for the E810 counters specifically to test your XOFF-burst theory.
This produced an interesting result: there was no XOFF/XON burst immediately before this NOP timeout.

From 08:42:30 through 08:43:07 the counters remained completely unchanged:

link_xon_rx.nic = 219631
link_xoff_rx.nic = 219631

There were also no RX drops or CRC errors.

Looking more closely at the PCAP showed another interesting pattern. Immediately before the blackout there is very heavy iSCSI traffic. The Linux TCP receive window becomes very small, and then there is approximately a 12-second period without useful iSCSI traffic before communication resumes.

So at least for this event I cannot see evidence that a concentrated burst of Ethernet PAUSE/XOFF frames immediately before the timeout caused the blackout.

I have now noticed another possible correlation which may be more interesting: the NOP events seem to coincide with backup activity.

For today's event:

08:43:07 first NOP timeout
08:43:10 recovered
08:43:24 second NOP timeout
08:43:27 recovered
08:43:53 Finished Backup of VM 304
08:43:54 Starting Backup of VM 305

VM 304 has five virtual disks on the ME5-backed stor1-vm storage. At the end of the Proxmox vzdump log, five fleecing images are removed before the backup is reported as finished:

removing (old) fleecing image 'local-vm:vm-304-fleece-0'
removing (old) fleecing image 'local-vm:vm-304-fleece-1'
removing (old) fleecing image 'local-vm:vm-304-fleece-2'
removing (old) fleecing image 'local-vm:vm-304-fleece-3'
removing (old) fleecing image 'local-vm:vm-304-fleece-4'
Finished Backup of VM 304

I do not want to conclude yet that removal of the fleecing images / backup cleanup is the cause. The individual cleanup lines in the vzdump task log do not contain timestamps, and an earlier NOP event also occurred while another backup was still in progress rather than right at its end.

However, the apparent correlation with backup/storage activity is interesting enough that I have extended the monitoring again.

In addition to the one-second NIC monitoring I am now recording the dm-0 block statistics for the ME5 LUN every second:

reads / sectors read / read time
writes / sectors written / write time
I/Os currently in progress
I/O time
weighted I/O time
discards
flush requests / flush time

The same logger also records:

XON/XOFF RX/TX
RX drops
CRC errors
MAC remote faults
RX/TX packets and bytes
RX allocation failures
TX busy/restart counters
softnet dropped/time_squeeze
NIC IRQ activity

The event-triggered script preserves the PCAPs, diagnostics and the one-second counter history when the kernel reports ISCSI_ERR_NOP_TIMEDOUT.

So the next occurrence should tell us whether there is a sudden block-I/O/flush spike or queue buildup immediately before the NOP timeout, whether the block-I/O counters stop progressing during the blackout, or whether the issue is still somewhere below that layer.

For now, the two useful findings from today's event seem to be:

1. Updating the HPE E810 firmware from NVM 4.91 to HPE NVM 5.00 did not eliminate the NOP timeout.
2. There was no concentrated XOFF/Pause burst immediately before this particular NOP timeout.

I'll leave the extended monitoring running and post the next captured event.

Thanks