Proxmox VE 9.2 – sporadic iSCSI NOP timeouts / DID_TRANSPORT_DISRUPTED with Dell PowerVault ME5 and Intel E810

I see that the target stops answering for like 12 seconds in that trace, did you reach out to Dell?

They usually have awesome support.

If you could reproduce it easily with another Version, I wouldn't rule out the iSCSI target not playing nice
 
Hi @huganega,

Thanks for the extra tracing. Some of it is useful, but I think the text you've put together is pointing you the wrong way.

The solid parts:

- The same stall shows up in two independent places: your I/O completions freeze for 12.2s, and the network gap from the target's keepalive to your FIN is 12.199s. This is really a single event.
- The pause-frame counters stay flat the whole time. No flow control on the NIC or the link, so it isn't that (as expected).
- ioerr jumping from 1 to 129 is just the 128 in-flight commands failing when the connection dropped. Also expected.

Here's the key point: the window that closed is the host's receive window. The host closed it, by not reading from the socket, so the array had no choice but to stop. It dropped to a few KB, and the ME5 correctly stopped sending and waited. The array was already throttled by the host before it could have done anything wrong, which puts the start of the stall on the host's receive side and clears the ME5. An actual iSCSI initiator/target hang looks different.

Where I'd be careful with your interpretation:

The D-state stacks show the block layer is stuck, but not why. Everything in them follows from commands not completing. Once the connection stops draining, the command queue fills, new I/O can't get a slot (your KVM threads), and flushes can't finish (the fsync workers). Those are what a stall causes, not what causes the stall. So "the block layer is stuck before the NOP timeout" is true, but it's the same stall.

The one thing we still don't have is what the real iscsi_q worker was doing. Your sample caught only the rescuer thread, and a rescuer sits idle in rescuer_thread() until it's needed, so it tells us nothing about the busy worker. However that happened, the fix is the same: don't sample for it. Have the kernel dump every blocked task at once.

To recap: the NOP timeout is a symptom, not the cause. The stall is on the host's receive side, not the ME5 or the link. What's left to figure out is which host layer stalls first: the block layer or the iSCSI/TCP path.

Add these to your trigger script so they fire when the queue hits the threshold:

Code: 
echo w > /proc/sysrq-trigger   # every task stuck waiting, dumped in one shot
echo l > /proc/sysrq-trigger   # what each CPU is doing right now

The first one is the most important one. It shows whether the thread servicing the socket is in the blocked list and what it's waiting on, which tells us whether the iSCSI path is stuck alongside the flush workers or is the thing they're all stuck behind. That's the question we need answered.

Please post the raw dumps, not a writeup... The polished summaries are harder to reason from, and twice now they've pointed at a cause the traces don't support. Raw output is what we need.

Let us know how the next one looks.

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox
 
  • Like
Reactions: Francois Baillargeon and Onslow
Another stall occurred on Sep 11 at 03:03:03.

As requested, I am attaching the raw data from the event without further filtering or interpretation.

Archive:
iscsi-forum-raw-2026-09-11_03-03-03.tar.gz

SHA256:
2b64f8cf9a9d944bceb7e02a350061616a867b18d032187cfc255ab5ea197993

Download:
https://1drv.ms/u/c/a605a90763ea0fad/IQBBmuNYdnDWQIdvemhuT_aZAWGpdJdFKf0nuKuZKYZ954w?e=8jR7l6

It contains the complete causal trace including SysRq-w/SysRq-l output and the full rolling packet capture set for the event.
 
Intriguing. Nothing in your data points to a root cause, and it doesn't look like the Dell. Our best guess is a Linux issue.

That said, we run a lot of iSCSI with PVE in the field and haven't seen any issues at customer sites. This suggests our integration may be avoiding it, which might offer a clue.

Can you enable this on your PVE node?
sysctl -w net.ipv4.tcp_shrink_window=1

You can revert it with:
sysctl -w net.ipv4.tcp_shrink_window=0


Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox
 
Do you have test lab where the same issue observed? I wanted to suggest to even try with Kernel 6.x, it's seems something specific with your hardware.
 
bbgeek17 said:
Intriguing. Nothing in your data points to a root cause, and it doesn't look like the Dell. Our best guess is a Linux issue.

That said, we run a lot of iSCSI with PVE in the field and haven't seen any issues at customer sites. This suggests our integration may be avoiding it, which might offer a clue.

Can you enable this on your PVE node?
sysctl -w net.ipv4.tcp_shrink_window=1

You can revert it with:
sysctl -w net.ipv4.tcp_shrink_window=0


Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox
Click to expand...

Enabled now:
net.ipv4.tcp_shrink_window = 1
I will leave the existing tracing and packet capture running unchanged and monitor for another occurrence.
 
longer said:
Do you have test lab where the same issue observed? I wanted to suggest to even try with Kernel 6.x, it's seems something specific with your hardware.
Click to expand...
Unfortunately, we do not have a separate test lab where the issue can be reproduced.


The problem is sporadic and only occurs on the production PVE host. We have not found a workload or test that can reliably trigger it, which makes controlled A/B testing difficult.


We have already seen the same issue with multiple PVE kernels on this host, so it does not appear to be limited to one specific current kernel build.


Testing an older 6.x kernel could still be useful as an additional comparison, but because the issue cannot be triggered on demand and the host is in production, we would have to run that kernel for some time to get a meaningful result.


At the moment we are also testing net.ipv4.tcp_shrink_window=1 as suggested in this thread, while leaving the existing tracing and packet capture unchanged.


If the issue still reproduces with that setting, trying a 6.x kernel would be one of the next useful comparisons.
 
  • Like
Reactions: longer
Hi,

I checked the weekend run in more detail.

net.ipv4.tcp_shrink_window=1 was enabled during the period from September 13, around 19:00, until September 14, around 06:00.

Interestingly, we did not see any iSCSI NOP timeouts during that period. However, the system was definitely not healthy: we experienced severe stalls and storage performance was extremely poor.

So at least in our case, enabling tcp_shrink_window=1 did not actually solve the underlying problem. It may have changed the TCP/iSCSI behaviour enough that the previous NOP timeout symptom disappeared, but the stalls and performance degradation were still very significant.

This also means that using the absence of NOP timeouts alone as an indication that the issue is resolved would be misleading.

Unfortunately, the rolling packet capture covering exactly that time window is no longer available, so I cannot compare the TCP window behaviour from that period packet-by-packet.

1789448157289.png
 
Hi @huganega,

Good news, this lines up with what we thought was going on.

Let me pin one thing down, though. With net.ipv4.tcp_shrink_window=1 set, are the timeouts and transport disconnects actually gone, or just less frequent? I want to know if it fully kills the problem in your title.

Just to be clear, we expected this knob only to affect disconnects. It is an important data point for understanding the root cause. It won't affect throughput. The slowness is a separate issue we can investigate after.

My read is that your array looks pretty sluggish. Is that a bare ME5224 running HDDs, not SSDs? Do you have an active support contract with Dell?


Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox
 
Hi,

during the test period with

net.ipv4.tcp_shrink_window=1

enabled, I did not observe any of the previous iSCSI NOP timeout events or transport disconnects.

So, during that test period, the disconnect issue from the thread title did not occur anymore.

However, the storage performance became too poor for us to leave net.ipv4.tcp_shrink_window=1 enabled in production. We therefore disabled it again. The performance issue still needs to be treated separately from the disconnect issue.

Regarding the ME5224 configuration:

The array uses a dynamic pool configuration. The current layout consists of:

  • 2 × 1.92 TB SAS SSDs as a RAID1 disk group
  • 14 × 2.4 TB 10K SAS HDDs in an ADAPT disk group
  • 1 × 1.92 TB SSD as global hot spare
  • 1 × 2.4 TB HDD as global hot spare
So this is not simply a bare 15-disk HDD RAID6 setup.

We also have an active support case open with Dell in parallel and are investigating the storage performance with them as well.

The Proxmox host is directly connected to both ME5224 controllers via 25 GbE iSCSI with multipath/ALUA.

Best regards,
Werner
 
Hi Werner,

Thanks for the confirmation.

I think this points squarely at Linux kernel behavior, not the array or the fabric. Slow storage makes it a lot easier to hit, but it isn't the cause, IMHO.

We have filed it in our dev queue for tracking and further investigation for a potential kernel fix. I'm confident they can track it down, but I can't promise timing since we don't see these sorts of issues in the field with our storage.

My advice is to drive it with your Dell team. They are Proxmox partners and hold your support contract, so they should be able to assign resources and push a fix upstream.

Best regards,
bbgeek17

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox
 
Hi,

thanks, that makes sense and also matches what we are currently seeing.

We will continue working with Dell in parallel. We already have an active support case open with them regarding the ME5224 performance and iSCSI behaviour.

Proxmox support has also suggested testing:

iscsi_tcp.recv_from_iscsi_q=Y

We enabled this yesterday and recreated both iSCSI sessions afterwards, so both paths are now running with the parameter active.

The next important test will be the backup window, because this is where the severe stalls and high in-guest I/O wait have been most reproducible.

We will monitor for:

  • iSCSI transport disconnects / NOP timeouts
  • D-state / hung tasks
  • storage stalls
  • VM I/O wait
  • backup duration and performance
I will post the results once we have data from the next backup run.

Thanks again for pushing this into the dev queue.

Best regards,
Werner
 
Hi Werner,

That's a solid advice. recv_from_iscsi_q=Y moves the receive path onto its own worker instead of running inline. If the problem is in the area where we think it is, this workaround should avoid the bug!

I didn't recommend it because I'd assumed the parameter was load-time only (TIL: it's a runtime module parameter) and we don't run this way in any of our deployments.

Let us know how it goes.

Best regards

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox
 
Good morning, everyone!

Lots of learning available from this forum thread, what a journey, Werner! You have a lot of patience and dedication to follow through with this troubleshooting. Very good sleuthing on BBGeek's part here, it makes sense that the NOP is symptomatic of a stall. The shrinking window problem has been quite elusive for us all it seems, but I am hopeful that the separate ISCSI worker helps it along.

I think @bbgeek17 and @longer are both right, though, this is perhaps a newer kernel issue: https://github.com/open-iscsi/open-iscsi/discussions/495

It seems to me that from that GitHub example and this thread, newer Linux kernels (e.g., 6.8+), certain default parameter behaviors in the open-iscsi driver can cause these subtle window issues. Setting a fixed or low value for tcp.window_size in /etc/iscsi/iscsid.conf overrides the Linux kernel’s native TCP autotuning capabilities. So, as shown in the GitHub thread, if the connection undergoes sudden latency spikes or heavy backup loads, the client's static window limit will choke the link, causing a gradual stall accompanied by TcpExtTCPWantZeroWindowAdv kernel alerts.

@huganega / Werner, if you're still experiencing issues, don't just revert to 6.8, that GitHub thread has too similar a complaint as yours. I'll try to figure out which version open-iscsi changed, but really the answer must be in these tcp.window_size settings (imo).

Thanks,


Triston
 
Ok @huganega , I think we've figured this out. It takes a combination of things to trigger:

- A link that's "fast" (i.e., 25GbE) relative to your CPU (Skylake or Cascade Lake)
- A backup load running at full queue depth
- An array that doesn't actively probe for TCP window updates

The bug is a TCP window deadlock in the iSCSI initiator, stemming from an interaction between the iSCSI driver and some interesting behavior in Linux TCP itself. Along the way, we may also have uncovered some questionable window handling in TCP.

We have a few patches in hand. It's going to take some time to test it, get it upstream, and get it into PVE. Once that's done, we'll post a technical write-up.

Short term, recv_from_iscsi_q=Y on the iscsi_tcp module should keep you out of it. tcp_shrink_window=1 also stops the disconnects, but that was just a diagnostic.

Cheers,
Blockbridge

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox
 
  • Like
Reactions: MarkusKo
Hi everyone,

thanks a lot for the detailed feedback and for digging into this further.

We now have two full backup days completed with:

iscsi_tcp.recv_from_iscsi_q=Y

enabled, and so far we have not seen any of the previous iSCSI NOP timeout events.

This is currently the only change related to the iSCSI receive path that remains enabled.

So from our side, the result after two backup cycles is:

  • no NOP timeouts
  • no iSCSI transport disconnects observed
  • backups completed
  • recv_from_iscsi_q=Y remains active on both iSCSI sessions
That is a very strong improvement compared to the previous behaviour.

The explanation around a TCP window deadlock in the iSCSI initiator also fits the symptoms we observed very well: the issue was most reproducible with 25 GbE iSCSI, under heavy backup load and full queue depth, while the system could otherwise operate normally for long periods.

Regarding the tcp.window_size discussion in iscsid.conf: we have not changed those settings yet. Since recv_from_iscsi_q=Y is currently giving us stable results, I would prefer to keep the test conditions unchanged for now and continue monitoring before introducing another variable.

We also tested net.ipv4.tcp_shrink_window=1 earlier. It did stop the NOP/disconnect symptom during the test period, but storage performance became too poor for us to leave that setting enabled in production. So we are treating that strictly as a diagnostic result.

For now we will continue running with:

iscsi_tcp.recv_from_iscsi_q=Y

and monitor the upcoming backup cycles for NOP timeouts, transport disconnects, D-state/hung tasks and storage stalls.

If this remains stable, it would be a very good indication that moving the iSCSI receive processing onto the separate worker successfully avoids the problematic code path.

Thanks again to everyone involved, and especially for pushing this towards a proper kernel fix. I am very interested in the technical write-up and the upstream patches once they are ready.

Best regards,
Werner
 
Update after another full backup cycle and a reboot to the latest Proxmox kernel.

The host is now running:

Proxmox VE 9.2
pve-manager 9.2.20
kernel 7.0.14-16-pve
Dell PowerVault ME5224
2x direct 25 GbE iSCSI
ALUA multipath

After the reboot, iscsi_tcp.recv_from_iscsi_q was back to its default value N.

I changed it to:

iscsi_tcp.recv_from_iscsi_q=Y
and then recreated both iSCSI sessions one path at a time, so both sessions were established after enabling the parameter.

Multipath was verified afterwards and remained healthy:

prio=50 status=active
sdk active ready running

prio=10 status=enabled
sdj active ready running
No other TCP/iSCSI tuning was changed for this test. In particular, tcp.window_size was left unchanged.

Immediately after the reboot and session recreation, the relevant TCP counters were all zero:

TcpExtTCPFromZeroWindowAdv 0
TcpExtTCPToZeroWindowAdv 0
TcpExtTCPWantZeroWindowAdv 0
After starting the VMs:

TcpExtTCPFromZeroWindowAdv 9
TcpExtTCPToZeroWindowAdv 5
TcpExtTCPWantZeroWindowAdv 5466
After the overnight PBS backup completed:

TcpExtTCPFromZeroWindowAdv 18
TcpExtTCPToZeroWindowAdv 10
TcpExtTCPWantZeroWindowAdv 868725
So during the backup window alone, TCPWantZeroWindowAdv increased by approximately 863k.

This is important because it shows that the TCP receive-window pressure is still clearly present under backup load. recv_from_iscsi_q=Y therefore does not appear to eliminate the underlying TCP window condition itself.

However, despite that large increase:

  • the backup completed successfully
  • there were no iSCSI NOP timeout events
  • there were no iSCSI transport disconnects
  • there were no session recovery events
  • there were no SCSI or I/O errors
  • multipath remained healthy for the entire backup
  • our NOP/event watcher did not create any new event captures
The kernel log filter for NOP timeouts, connection/transport failures, session recovery, task aborts and SCSI/I/O errors returned no events.

There is one additional observation which may be relevant.

Our debugging periodically triggers SysRq blocked-task/all-CPU snapshots during high storage pressure. In one snapshot at 00:00:28, an iSCSI transmit worker was caught in D-state:

task:kworker/u259:3 state:D
Workqueue: iscsi_q_3 iscsi_xmitworker [libiscsi]

iscsi_xmit_task
schedule
At the same time, the receive side was running via the separate workqueue introduced by recv_from_iscsi_q=Y:

Workqueue: iscsi_q_3 iscsi_sw_tcp_recv_data_work [iscsi_tcp]
RIP: iscsi_tcp_segment_done
...
tcp_read_sock
iscsi_sw_tcp_recv_data_work
There were no kernel messages such as:

task ... blocked for more than X seconds
The phrase only appears in the grep command itself, not as an actual kernel warning.

So my current interpretation is:

iscsi_tcp.recv_from_iscsi_q=Y does not remove the TCP receive-window pressure, and transiently blocked iSCSI workers can still be observed under heavy backup load.

However, since enabling it and recreating the sessions, that condition has so far no longer developed into the previous long iSCSI stall / NOP timeout / transport disconnect.

We have now completed multiple backup cycles with recv_from_iscsi_q=Y without reproducing the original NOP timeout.

I will keep the current configuration unchanged and continue monitoring further backup cycles.
 
You must log in or register to reply here.
Top Bottom
Back