Putting this here in case anyone else is affected by this issue.
We've been having an issues with VM performance after upgrading our cluster to PVE9 on kernel 7.0.2-6-pve. This has been happening for a few of months and we finally managed to pin it down.
Symptoms on the guest varied, it wasn't noticeable on some VMs, other VMs saw significant CPU usage increases. I was mainly noticeable on when the VM was under load. On the worst affected VMs we would notice CPU usage increase to 100% for 2-3s every 60s in top/htop. When checking on the host we noticed that this correlated when kvm-nx-lpage-re thread started processing.
The workaround was to disable nx_huge_pages_recovery_ratio.
We have not seen any issues since implementing the workaround. The fix is to roll back to kernel 6.17.x, but we have not tested this. It does not look like this bug has been logged upstream yet.
Analysis of the kernel code by Claude points at this as a bug in the kernel introduced in 6.19+.
The defect: kvm_recover_nx_huge_pages() sizes its loop from a per-MMU-type counter, kvm->arch.possible_nx_huge_pages[mmu_type].nr_pages. The unaccount path does not honour the type:
static void unaccount_nx_huge_page(struct kvm *kvm, struct kvm_mmu_page *sp)
{
sp->nx_huge_page_disallowed = false;
untrack_possible_nx_huge_page(kvm, sp, KVM_SHADOW_MMU); /* hardcoded */
}
It takes no mmu_type and always passes KVM_SHADOW_MMU. So on a TDP MMU guest, which is every guest on current Intel and AMD hardware, recovering a page decrements the shadow counter instead of the TDP one. The corrective decrement in tdp_mmu_unlink_sp() never
runs, because unaccount_nx_huge_page() cleared sp->nx_huge_page_disallowed on the line above and it returns early.
Consequence: nr_pages[KVM_TDP_MMU] is incremented on every split and never decremented, growing monotonically for the life of the guest. The loop bound derived from it grows without limit, so the pass exits only when the list empties, while vCPUs re-fault and re-split pages onto it as it runs. Measured on vm: 305,168 calls to kvm_tdp_mmu_zap_possible_nx_huge_page in 130 s where the design intends 4, a factor of 76,292.
nx_lpage_splits is maintained correctly, so the debugfs gauge looks sane while the real driver is a hidden counter not exposed anywhere. Cost therefore runs inverse to the visible split count and tracks guest age instead. The worker is a vhost_task inside the QEMU thread group, so its time bills to the guest as proxmox.qemu.cpu, and it is bursty (a few seconds at 99% of a core, then ~50 s asleep), which cumulative sampling misses entirely.
Fix: roll back to proxmox-kernel-6.17 (6.17.13-21, pve-enterprise), which predates the per-MMU-type array and has mitigation parity with 7.0.14. Needs a maintenance window.
We've been having an issues with VM performance after upgrading our cluster to PVE9 on kernel 7.0.2-6-pve. This has been happening for a few of months and we finally managed to pin it down.
Symptoms on the guest varied, it wasn't noticeable on some VMs, other VMs saw significant CPU usage increases. I was mainly noticeable on when the VM was under load. On the worst affected VMs we would notice CPU usage increase to 100% for 2-3s every 60s in top/htop. When checking on the host we noticed that this correlated when kvm-nx-lpage-re thread started processing.
Code:
top -H -b -p $(cat /var/run/qemu-server/<VMID>.pid)
top - 12:53:36 up 3 days, 14:29, 1 user, load average: 22.37, 21.69, 22.33
Threads: 15 total, 5 running, 10 sleeping, 0 stopped, 0 zombie
%Cpu(s): 20.5 us, 26.0 sy, 0.0 ni, 52.7 id, 0.4 wa, 0.0 hi, 0.5 si, 0.0 st
MiB Mem : 772546.6 total, 390061.2 free, 219608.5 used, 169506.2 buff/cache
MiB Swap: 1022.0 total, 1022.0 free, 0.0 used. 552938.1 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
48875 root 20 0 20.8g 15.5g 11288 R 99.9 2.1 6,10 CPU 2/KVM
48876 root 20 0 20.8g 15.5g 11288 R 99.9 2.1 6,16 CPU 3/KVM
48874 root 20 0 20.8g 15.5g 11288 R 99.7 2.1 6,10 CPU 1/KVM
49077 root 20 0 20.8g 15.5g 11288 R 99.0 2.1 15:43.10 kvm-nx-lpage-re
48873 root 20 0 20.8g 15.5g 11288 R 56.8 2.1 222:55.52 CPU 0/KVM
48872 root 20 0 20.8g 15.5g 11288 S 17.6 2.1 177:53.83 vhost-48832
48832 root 20 0 20.8g 15.5g 11288 S 0.0 2.1 1:16.95 kvm
48833 root 20 0 20.8g 15.5g 11288 S 0.0 2.1 0:00.00 call_rcu
48879 root 20 0 20.8g 15.5g 11288 S 0.0 2.1 0:00.00 vnc_worker
974602 root 20 0 20.8g 15.5g 11288 S 0.0 2.1 0:00.00 worker
975530 root 20 0 20.8g 15.5g 11288 S 0.0 2.1 0:00.00 worker
975531 root 20 0 20.8g 15.5g 11288 S 0.0 2.1 0:00.00 worker
975532 root 20 0 20.8g 15.5g 11288 S 0.0 2.1 0:00.00 worker
975533 root 20 0 20.8g 15.5g 11288 S 0.0 2.1 0:00.00 worker
975534 root 20 0 20.8g 15.5g 11288 S 0.0 2.1 0:00.00 workerThe workaround was to disable nx_huge_pages_recovery_ratio.
Code:
echo 0 > /sys/module/kvm/parameters/nx_huge_pages_recovery_ratioWe have not seen any issues since implementing the workaround. The fix is to roll back to kernel 6.17.x, but we have not tested this. It does not look like this bug has been logged upstream yet.
Analysis of the kernel code by Claude points at this as a bug in the kernel introduced in 6.19+.
The defect: kvm_recover_nx_huge_pages() sizes its loop from a per-MMU-type counter, kvm->arch.possible_nx_huge_pages[mmu_type].nr_pages. The unaccount path does not honour the type:
static void unaccount_nx_huge_page(struct kvm *kvm, struct kvm_mmu_page *sp)
{
sp->nx_huge_page_disallowed = false;
untrack_possible_nx_huge_page(kvm, sp, KVM_SHADOW_MMU); /* hardcoded */
}
It takes no mmu_type and always passes KVM_SHADOW_MMU. So on a TDP MMU guest, which is every guest on current Intel and AMD hardware, recovering a page decrements the shadow counter instead of the TDP one. The corrective decrement in tdp_mmu_unlink_sp() never
runs, because unaccount_nx_huge_page() cleared sp->nx_huge_page_disallowed on the line above and it returns early.
Code:
┌───────────────────────────┬─────────────────────────────┬───────────┐
│ counter │ actual │ intended │
├───────────────────────────┼─────────────────────────────┼───────────┤
│ kvm->stat.nx_lpage_splits │ -1 │ -1 │
├───────────────────────────┼─────────────────────────────┼───────────┤
│ nr_pages[KVM_TDP_MMU] │ unchanged │ -1 │
├───────────────────────────┼─────────────────────────────┼───────────┤
│ nr_pages[KVM_SHADOW_MMU] │ -1 (underflows toward 2^64) │ unchanged │
└───────────────────────────┴─────────────────────────────┴───────────┘Consequence: nr_pages[KVM_TDP_MMU] is incremented on every split and never decremented, growing monotonically for the life of the guest. The loop bound derived from it grows without limit, so the pass exits only when the list empties, while vCPUs re-fault and re-split pages onto it as it runs. Measured on vm: 305,168 calls to kvm_tdp_mmu_zap_possible_nx_huge_page in 130 s where the design intends 4, a factor of 76,292.
nx_lpage_splits is maintained correctly, so the debugfs gauge looks sane while the real driver is a hidden counter not exposed anywhere. Cost therefore runs inverse to the visible split count and tracks guest age instead. The worker is a vhost_task inside the QEMU thread group, so its time bills to the guest as proxmox.qemu.cpu, and it is bursty (a few seconds at 99% of a core, then ~50 s asleep), which cumulative sampling misses entirely.
Fix: roll back to proxmox-kernel-6.17 (6.17.13-21, pve-enterprise), which predates the per-MMU-type array and has mitigation parity with 7.0.14. Needs a maintenance window.
