Severe system freeze with NFS on Proxmox 9 running kernel 6.14.8-2-pve when mounting NFS shares

[Davidegb]

@hybridencounter Thanks for your data. Let's see if with data from both cases we can narrow down the issue so developers can finally fix it

Update — further investigation, more data for developers

After more testing today, I can refine the diagnosis considerably.

The bug requires NFS in the path.I tried to reproduce the issue with sustained write load inside the NAS VM itself —copying from the NAS VM's local ZFS pool to a secondary virtual disk backed by the host's NVMe, and 4× parallel dd — over 80 GB without a single issue. The VM handles local I/O fine under heavy load. The deadlock only triggers when NFS is involved. So this is not a pure I/O regression: it sits at the intersection of pve-qemu-kvm 11.0.0-4 + NFS traffic + kernel 6.17.13.

Kernel 7.0.6 is not a universal workaround.It resolved my situation, but @hybridencounter reports it did not help him — only the pve-qemu-kvm downgrade did. So the reliable fix is the downgrade, not the kernel upgrade.

Confirmed fix (both of us):

apt install pve-qemu-kvm=11.0.0-3
apt-mark hold pve-qemu-kvm

Restart affected VMs (full system reboot not strictly required).

For developers:The regression is tightly bisectable: 11.0.0-3 = stable, 11.0.0-4 = broken, confirmed by at least two independent users with different setups. The 11.0.0-4 changelog mentions backported fixes for aio=native, virtio-blk and aio. One of those commits appears to affect the NFS path specifically — possibly related to how the guest virtual network or disk stack interacts with sustained NFS write traffic. Local VM I/O is unaffected, which suggests the trigger involves the virtual network layer, not just disk I/O.

Call trace from the host dmesg at the moment of deadlock (kernel 6.17.13-13, pve-qemu-kvm 11.0.0-4):

nfs: server nas not responding, still trying
INFO: task CPU x/KVM blocked for more than 122 seconds.
  nfs_wb_folio → folio_wait_writeback
    → __folio_split → migrate_pages_batch
      → kvm_mmu_faultin_pfn → npf_interception [kvm_amd]
INFO: task worker:xxxx is blocked on an rw-semaphore,
      but the owner is not found.  (×N workers)

Same trace reproduced consistently on 6.17.13-1, -11, and -13. Zero occurrences with 11.0.0-3 under identical load.

Hope this helps narrow it down.

 

[Davidegb]

Update 2 — deeper investigation, the rabbit hole goes further
More testing today revealed something unexpected that significantly refines the diagnosis.
Root cause: two independent but chained bugs.
After extensive testing across multiple kernels and configurations, here is the complete picture.

---

Bug 1 — vCPU starvation in pve-qemu-kvm 11.0.0-4
Under sustained NFS write load, with the NFS server running as a KVM VM on the same Proxmox host, pve-qemu-kvm 11.0.0-4 causes vCPU starvation inside the NAS VM. The NAS VM's kernel detects that one of its vCPUs has received zero CPU ticks during the current RCU grace period, the RCU kthread starves, and the NAS stops responding to NFS requests. This is not a network issue — the NAS dies from the inside.
From the NAS VM dmesg (Debian kernel 6.12.90, guest OS):

rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:
rcu:     2-...!: (0 ticks this GP) idle=e174/1/0x4000000000000000
rcu: rcu_preempt kthread starved for 7735 jiffies!
rcu:     Unless rcu_preempt kthread gets sufficient CPU time,
         OOM is now expected behavior.

Bug 2 — NFS client deadlock in kernel 6.17.x
When the NFS server stops responding, the host's 6.17.x NFS client deadlocks via the memory compactor path:

nfs: server nas not responding, still trying
INFO: task CPU x/KVM blocked for more than 122 seconds.
  nfs_wb_folio → folio_wait_writeback
    → __folio_split → migrate_pages_batch
      → kvm_mmu_faultin_pfn → npf_interception [kvm_amd]
INFO: task worker:xxxx is blocked on an rw-semaphore,
      but the owner is not found.  (× N QEMU workers)

The host has NFS folios stuck in writeback that will never complete. The memory compactor tries to migrate them, blocks forever, KVM enters uninterruptible sleep (state D), and all QEMU worker threads pile up on the same dead rw-semaphore. Notably, processes completely unrelated to NFS (e.g. Grafana in an LXC) also end up in state D — not because they touch NFS, but because the memory compactor is stuck and any process needing memory allocation at that moment gets blocked too.
(I used Claude AI for dmesg analysis)

---

The bug requires a cold NAS VM start.
As I posted just before, I tried to isolate whether NFS was actually necessary by running local I/O first — ~100 GB cp followed by 4× parallel dd (80 GB total, ~370 MB/s) entirely within the NAS VM, from ZFS to a secondary virtual disk backed by the host's NVMe. No issues at all.
Then, in the same session without restarting the NAS VM, I launched the NFS copy test. Surprisingly, it completed cleanly.
I repeated this test in both 6.17.13-1 and 6.17.13-13. I was able to copy 100 GB and dd 80 GB through NFS without error on both kernels + pve-qemu-kvm 11.0.0-4 after previous local VM I/O stress test!

However, also with both kernels, restarting the NAS VM and launching NFS copy directly failed again within 10 GB, same deadlock as before.
It seems the bug is significantly less likely to trigger after prior local I/O load on the NAS VM. However, this was not an exhaustive test — it is possible that the local I/O only delays or reduces the probability of the bug rather than preventing it entirely. A cold VM start with NFS as the first sustained I/O load triggers it reliably and quickly (within ~10 GB).

---

Kernel 7.0.6 results
Ran the same cold-start protocol on kernel 7.0.6-2-pve with pve-qemu-kvm 11.0.0-4:

• NAS VM started fresh, no prior local I/O
• ~100 GB cp to NFS mount
• 4× parallel dd, 80 GB total
• NAS dmesg monitored in real time throughout

Result: completely clean. Zero RCU stalls, zero errors in NAS dmesg. Host stable throughout.
Kernel 7.0.6 prevents Bug 1 entirely — the vCPU starvation does not occur. Since Bug 1 never triggers, Bug 2 has no opportunity to manifest. Whether kernel 7.0.6 would handle Bug 2 gracefully if Bug 1 were triggered by other means was not tested.

---

Updated summary:

Kernel	pve-qemu-kvm	Cold VM + NFS load	Result
6.17.13-x (any)	11.0.0-3	✓	Stable
6.17.13-x (any)	11.0.0-4	✗	Deadlock <10 GB
7.0.6-2	11.0.0-4	✓	Stable, >180 GB clean

Confirmed fix (verified by at least two independent users):

apt install pve-qemu-kvm=11.0.0-3
apt-mark hold pve-qemu-kvm

Restart affected VMs. Full system reboot not strictly required.
Alternative workaround: upgrade to kernel 7.0.6-2-pve. Confirmed stable in this setup, but did not resolve the issue for all affected users — see previous @hybridencounter reply in this thread.

Full dmesg logs from both Proxmox host and NAS VM are available on request.

---

Note: Bug 1 (vCPU starvation) requires the NFS server to be a KVM VM on the same Proxmox host with iothread enabled. Users with an external NAS are likely only affected by Bug 2, triggered by whatever causes their NFS server to become temporarily unresponsive.

Correction to the note at the bottom: it is not confirmed that iothread=1 on the NAS VM is required to trigger Bug 1. This was assumed based on prior iothread-related issues with QEMU 11.0, but was not explicitly tested. The NAS VM in our setup has iothread enabled, but whether disabling it would prevent the vCPU starvation is unknown. I will try it and post results.