Hi
@huganega,
Thanks for the extra tracing. Some of it is useful, but I think the text you've put together is pointing you the wrong way.
The solid parts:
- The same stall shows up in two independent places: your I/O completions freeze for 12.2s, and the network gap from the target's keepalive to your FIN is 12.199s. This is really a single event.
- The pause-frame counters stay flat the whole time. No flow control on the NIC or the link, so it isn't that (as expected).
- ioerr jumping from 1 to 129 is just the 128 in-flight commands failing when the connection dropped. Also expected.
Here's the key point: the window that closed is the host's receive window. The host closed it, by not reading from the socket, so the array had no choice but to stop. It dropped to a few KB, and the ME5 correctly stopped sending and waited. The array was already throttled by the host before it could have done anything wrong, which puts the start of the stall on the host's receive side and clears the ME5. An actual iSCSI initiator/target hang looks different.
Where I'd be careful with your interpretation:
The D-state stacks show the block layer is stuck, but not why. Everything in them follows from commands not completing. Once the connection stops draining, the command queue fills, new I/O can't get a slot (your KVM threads), and flushes can't finish (the fsync workers). Those are what a stall causes, not what causes the stall. So "the block layer is stuck before the NOP timeout" is true, but it's the same stall.
The one thing we still don't have is what the real iscsi_q worker was doing. Your sample caught only the rescuer thread, and a rescuer sits idle in rescuer_thread() until it's needed, so it tells us nothing about the busy worker. However that happened, the fix is the same: don't sample for it. Have the kernel dump every blocked task at once.
To recap: the NOP timeout is a symptom, not the cause. The stall is on the host's receive side, not the ME5 or the link. What's left to figure out is which host layer stalls first: the block layer or the iSCSI/TCP path.
Add these to your trigger script so they fire when the queue hits the threshold:
Code:
echo w > /proc/sysrq-trigger # every task stuck waiting, dumped in one shot
echo l > /proc/sysrq-trigger # what each CPU is doing right now
The first one is the most important one. It shows whether the thread servicing the socket is in the blocked list and what it's waiting on, which tells us whether the iSCSI path is stuck alongside the flush workers or is the thing they're all stuck behind. That's the question we need answered.
Please post the raw dumps, not a writeup... The polished summaries are harder to reason from, and twice now they've pointed at a cause the traces don't support. Raw output is what we need.
Let us know how the next one looks.
Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox
