Hello all,
I'm hitting a recurring NFSv4.2 client-side failure on a PVE host. Three incidents over roughly three weeks. The most recent one produced a kernel oops with a corrupted return address on the kernel stack, inside the sunrpc receive path. After the oops the mount is dead, every process touching it goes into uninterruptible D-state, and eventually systemd itself stops responding to D-Bus (systemctl and journalctl hang). Only a host reboot recovers it.
The NFS server is a VM (with truenas) on the same host and stays perfectly healthy and it's web UI is responsive and restarting the NFS service on the server side does not unblock the client.
Looking for input on whether this matches a known sunrpc/skb issue, or whether I should be treating it as hardware.
Host
-Proxmox VE 9.x, kernel 7.0.14-12-pve (latest available in repo)
-ASRock Rack W680 WS, BIOS 21.12 (10/25/2024)
-Intel i5-13500
-128 GB DDR5 ECC
-Root on ZFS mirror, second ZFS mirror (NVMe) for application data
-Broadcom 9500-16i HBA passed through to the storage VM (VFIO)
Storage VM
- TrueNAS SCALE 25.10, 32 GB RAM, virtio-net on vmbr0
- Owns the HBA and a 12-drive pool (ca. 211 TiB), exports one dataset over NFS
- Host mounts that export back over the virtual bridge so the NFS traffic never leaves the box
Mount options as reported by /proc/mounts
Mounted via a systemd automount unit. Docker has a drop-in ordering dependency on the .automount unit.
Workload
This is the part I suspect is relevant, because it's unusual. The host runs a workload that keeps roughly 1100 concurrent file streams open against the NFS mount, generating a sustained, very high rate of small RPC operations, well above what a typical NFS client produces.
Three minutes later:
My thoughts
So...RIP and RSP are about 1 KB apart and both point into the same kernel stack region. So the CPU tried to execute from the stack, which is NX-mapped, and faulted. The Code: dump at RIP is not instructions, it looks like two kernel pointers followed by a run of zeroes. That reads to me like a return address on the stack got overwritten and ret jumped into data. The corruption happens during a 4 KB copy (RAX/RDX/R14/R15 all 0x1000) out of the TCP receive buffer into the RPC buffer. The worker dying with IRQs disabled is what leaves the socket unserviced, which is why nothing recovers afterwards.
Aftermath:
- rpciod/xprtiod kworker stuck in D-state indefinitely, same PID for hours
- Load average climbs to ~74, essentially all of it D-state processes
- systemctl daemon-reload and systemctl list-jobs hang (PID 1 not answering D-Bus)
- New SSH logins hang at authentication, presumably PAM waiting on logind
- Web UI, containers not touching the mount, and the storage VM all keep working
- Restarting the NFS service on the server side does not release the stuck worker — consistent with the worker being dead rather than blocked
- ZFS is unaffected: no CKSUM/READ/WRITE errors on either pool across three hard reboots
Earlier incidents:
Two prior incidents had the same shape. The rpciod in D-state, hang cascading through Docker, host reboot required but no oops was logged, or at least I didn't capture one. Stack on one of those was in xs_tcp_send_request (send side rather than receive side).
Already ruled out:
- nconnect - I initially suspected it, but the first incident predates enabling it. Not the trigger.
- Server-side failure - the storage VM is responsive and healthy during every incident.
- ZFS pool health - clean, no errors, scrubs pass.
- Kernel version - 7.0.14-12-pve is the newest in the repo, so there's nothing to upgrade to.
- RAM under load - 6-hour stressapptest run over 103 GB(966 TB transferred, 45,8 GB/s) completed with zero incidents and zero corrected ECC errors in EDAC. (Full memtest86+ over all 128 GB still pending; the board has no BMC so it needs physical console access. -> jetKVM on its way :-D )
Questions:
1. Does this stack signature match anything known in sunrpc or the skb/iter path on this kernel? I couldn't find a matching report.
2. Is a hard NFS mount to a guest on the same host over vmbr0 a configuration anyone else is running at high RPC rates? I'd like to know if the loopback-through-bridge topology is a factor.
3. Would switching the storage VM's NIC from virtio-net to e1000 be a sensible next isolation step? The oops is in the receive path immediately after the skb copy, so a driver-side buffer issue seems worth excluding — at the cost of throughput.
4. Anything else worth capturing before the next occurrence? I've set up a watchdog that mails me on kernel oops, nfs: server not responding, or any rpciod/nfs process sitting in D-state for more than six minutes, so I should now catch the next one within two minutes rather than two hours.
Happy to provide any additional logs or run diagnostics. Thanks for reading
I'm hitting a recurring NFSv4.2 client-side failure on a PVE host. Three incidents over roughly three weeks. The most recent one produced a kernel oops with a corrupted return address on the kernel stack, inside the sunrpc receive path. After the oops the mount is dead, every process touching it goes into uninterruptible D-state, and eventually systemd itself stops responding to D-Bus (systemctl and journalctl hang). Only a host reboot recovers it.
The NFS server is a VM (with truenas) on the same host and stays perfectly healthy and it's web UI is responsive and restarting the NFS service on the server side does not unblock the client.
Looking for input on whether this matches a known sunrpc/skb issue, or whether I should be treating it as hardware.
Host
-Proxmox VE 9.x, kernel 7.0.14-12-pve (latest available in repo)
-ASRock Rack W680 WS, BIOS 21.12 (10/25/2024)
-Intel i5-13500
-128 GB DDR5 ECC
-Root on ZFS mirror, second ZFS mirror (NVMe) for application data
-Broadcom 9500-16i HBA passed through to the storage VM (VFIO)
Storage VM
- TrueNAS SCALE 25.10, 32 GB RAM, virtio-net on vmbr0
- Owns the HBA and a 12-drive pool (ca. 211 TiB), exports one dataset over NFS
- Host mounts that export back over the virtual bridge so the NFS traffic never leaves the box
Mount options as reported by /proc/mounts
Code:
172.30.69.59:/mnt/DATA/Media /mnt/DATA/Media nfs4 rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,fatal_neterrors=none,proto=tcp,nconnect=4,timeo=600,retrans=2,sec=sys,clientaddr=172.30.69.56,local_lock=none,addr=172.30.69.59Workload
This is the part I suspect is relevant, because it's unusual. The host runs a workload that keeps roughly 1100 concurrent file streams open against the NFS mount, generating a sustained, very high rate of small RPC operations, well above what a typical NFS client produces.
Code:
kernel tried to execute NX-protected page - exploit attempt? (uid: 0)
BUG: unable to handle page fault for address: ffffcb0dcf6ebd68
#PF: supervisor instruction fetch in kernel mode
#PF: error_code(0x0011) - permissions violation
PGD 100000067 P4D 100000067 PUD 1008c9067 PMD 136b21067 PTE 80000017b8ef3163
Oops: Oops: 0011 [#1] SMP NOPTI
CPU: 16 UID: 0 PID: 1754485 Comm: kworker/u80:13 Tainted: P O 7.0.14-12-pve #1 PREEMPT(lazy)
Tainted: [P]=PROPRIETARY_MODULE, [O]=OOT_MODULE
Hardware name: W680 WS/W680 WS, BIOS 21.12 10/25/2024
Workqueue: xprtiod xs_stream_data_receive_workfn [sunrpc]
RIP: 0010:0xffffcb0dcf6ebd68
Code: ff ff 00 cc cd c1 b0 8a ff ff 10 76 22 c0 b0 8a ff ff 0c 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <02> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 14 cd 29 c9 8a
RSP: 0018:ffffcb0dcf6eb988 EFLAGS: 00010286
RAX: 0000000000001000 RBX: 0000000000006f94 RCX: 0000000000000000
RDX: 0000000000001000 RSI: ffff8abcbf6d906c RDI: ffff8ac7e7404000
RBP: ffffcb0dcf6eba18 R08: ffff8abcbf6d806c R09: 0000000000000000
R10: 0000000000000000 R11: ffff8ac6278a5900 R12: ffffcb0dcf6ebd68
R13: ffff8ac929cd1400 R14: 0000000000001000 R15: 0000000000001000
FS: 0000000000000000(0000) GS:ffff8ad01fb0e000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: ffffcb0dcf6ebd68 CR3: 000000142b387006 CR4: 0000000000f72ef0
PKRU: 55555554
Call Trace:
<TASK>
? _copy_to_iter+0x27f/0x610
? __ip_queue_xmit+0x19b/0x4e0
? __check_object_size+0xb4/0x270
? __pfx_simple_copy_to_iter+0x10/0x10
simple_copy_to_iter+0x3e/0x70
__skb_datagram_iter+0x1b8/0x2f0
? __pfx_simple_copy_to_iter+0x10/0x10
skb_copy_datagram_iter+0x39/0xc0
tcp_recvmsg_locked+0x836/0xb00
? __tcp_send_ack.part.0+0xf3/0x1f0
tcp_recvmsg+0x83/0x1f0
inet_recvmsg+0x51/0x130
? apparmor_socket_recvmsg+0x64/0x80
? security_socket_recvmsg+0x70/0x1e0
sock_recvmsg+0xb6/0xc0
xs_sock_recvmsg.constprop.0+0x2c/0xa0 [sunrpc]
xs_read_stream_request.constprop.0+0x255/0x4f0 [sunrpc]
xs_read_stream.constprop.0+0x2b3/0x450 [sunrpc]
xs_stream_data_receive_workfn+0x71/0x160 [sunrpc]
process_one_work+0x1a9/0x3c0
worker_thread+0x1b8/0x360
? __pfx_worker_thread+0x10/0x10
kthread+0xf7/0x130
? __pfx_kthread+0x10/0x10
ret_from_fork+0x2da/0x3a0
? __pfx_kthread+0x10/0x10
ret_from_fork_asm+0x1a/0x30
</TASK>
[...]
CR2: ffffcb0dcf6ebd68
---[ end trace 0000000000000000 ]---
note: kworker/u80:13[1754485] exited with irqs disabledThree minutes later:
Code:
nfs: server 172.30.69.59 not responding, still tryingMy thoughts
So...RIP and RSP are about 1 KB apart and both point into the same kernel stack region. So the CPU tried to execute from the stack, which is NX-mapped, and faulted. The Code: dump at RIP is not instructions, it looks like two kernel pointers followed by a run of zeroes. That reads to me like a return address on the stack got overwritten and ret jumped into data. The corruption happens during a 4 KB copy (RAX/RDX/R14/R15 all 0x1000) out of the TCP receive buffer into the RPC buffer. The worker dying with IRQs disabled is what leaves the socket unserviced, which is why nothing recovers afterwards.
Aftermath:
- rpciod/xprtiod kworker stuck in D-state indefinitely, same PID for hours
- Load average climbs to ~74, essentially all of it D-state processes
- systemctl daemon-reload and systemctl list-jobs hang (PID 1 not answering D-Bus)
- New SSH logins hang at authentication, presumably PAM waiting on logind
- Web UI, containers not touching the mount, and the storage VM all keep working
- Restarting the NFS service on the server side does not release the stuck worker — consistent with the worker being dead rather than blocked
- ZFS is unaffected: no CKSUM/READ/WRITE errors on either pool across three hard reboots
Earlier incidents:
Two prior incidents had the same shape. The rpciod in D-state, hang cascading through Docker, host reboot required but no oops was logged, or at least I didn't capture one. Stack on one of those was in xs_tcp_send_request (send side rather than receive side).
Already ruled out:
- nconnect - I initially suspected it, but the first incident predates enabling it. Not the trigger.
- Server-side failure - the storage VM is responsive and healthy during every incident.
- ZFS pool health - clean, no errors, scrubs pass.
- Kernel version - 7.0.14-12-pve is the newest in the repo, so there's nothing to upgrade to.
- RAM under load - 6-hour stressapptest run over 103 GB(966 TB transferred, 45,8 GB/s) completed with zero incidents and zero corrected ECC errors in EDAC. (Full memtest86+ over all 128 GB still pending; the board has no BMC so it needs physical console access. -> jetKVM on its way :-D )
Questions:
1. Does this stack signature match anything known in sunrpc or the skb/iter path on this kernel? I couldn't find a matching report.
2. Is a hard NFS mount to a guest on the same host over vmbr0 a configuration anyone else is running at high RPC rates? I'd like to know if the loopback-through-bridge topology is a factor.
3. Would switching the storage VM's NIC from virtio-net to e1000 be a sensible next isolation step? The oops is in the receive path immediately after the skb copy, so a driver-side buffer issue seems worth excluding — at the cost of throughput.
4. Anything else worth capturing before the next occurrence? I've set up a watchdog that mails me on kernel oops, nfs: server not responding, or any rpciod/nfs process sitting in D-state for more than six minutes, so I should now catch the next one within two minutes rather than two hours.
Happy to provide any additional logs or run diagnostics. Thanks for reading