KVM NX huge page recovery burns a core per guest on kernel 7.0.x

[waynej]

Putting this here in case anyone else is affected by this issue.

We've been having an issues with VM performance after upgrading our cluster to PVE9 on kernel 7.0.2-6-pve. This has been happening for a few of months and we finally managed to pin it down.

Symptoms on the guest varied, it wasn't noticeable on some VMs, other VMs saw significant CPU usage increases. I was mainly noticeable on when the VM was under load. On the worst affected VMs we would notice CPU usage increase to 100% for 2-3s every 60s in top/htop. When checking on the host we noticed that this correlated when kvm-nx-lpage-re thread started processing.

top -H -b -p $(cat /var/run/qemu-server/<VMID>.pid)

top - 12:53:36 up 3 days, 14:29,  1 user,  load average: 22.37, 21.69, 22.33
Threads:  15 total,   5 running,  10 sleeping,   0 stopped,   0 zombie
%Cpu(s): 20.5 us, 26.0 sy,  0.0 ni, 52.7 id,  0.4 wa,  0.0 hi,  0.5 si,  0.0 st
MiB Mem : 772546.6 total, 390061.2 free, 219608.5 used, 169506.2 buff/cache     
MiB Swap:   1022.0 total,   1022.0 free,      0.0 used. 552938.1 avail Mem

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND                                                                                                                                                                                                 
  48875 root      20   0   20.8g  15.5g  11288 R  99.9   2.1      6,10 CPU 2/KVM                                                                                                                                                                                               
  48876 root      20   0   20.8g  15.5g  11288 R  99.9   2.1      6,16 CPU 3/KVM                                                                                                                                                                                               
  48874 root      20   0   20.8g  15.5g  11288 R  99.7   2.1      6,10 CPU 1/KVM                                                                                                                                                                                               
  49077 root      20   0   20.8g  15.5g  11288 R  99.0   2.1  15:43.10 kvm-nx-lpage-re                                                                                                                                                                                         
  48873 root      20   0   20.8g  15.5g  11288 R  56.8   2.1 222:55.52 CPU 0/KVM                                                                                                                                                                                               
  48872 root      20   0   20.8g  15.5g  11288 S  17.6   2.1 177:53.83 vhost-48832                                                                                                                                                                                             
  48832 root      20   0   20.8g  15.5g  11288 S   0.0   2.1   1:16.95 kvm                                                                                                                                                                                                     
  48833 root      20   0   20.8g  15.5g  11288 S   0.0   2.1   0:00.00 call_rcu                                                                                                                                                                                               
  48879 root      20   0   20.8g  15.5g  11288 S   0.0   2.1   0:00.00 vnc_worker                                                                                                                                                                                             
 974602 root      20   0   20.8g  15.5g  11288 S   0.0   2.1   0:00.00 worker                                                                                                                                                                                                 
 975530 root      20   0   20.8g  15.5g  11288 S   0.0   2.1   0:00.00 worker                                                                                                                                                                                                 
 975531 root      20   0   20.8g  15.5g  11288 S   0.0   2.1   0:00.00 worker                                                                                                                                                                                                 
 975532 root      20   0   20.8g  15.5g  11288 S   0.0   2.1   0:00.00 worker                                                                                                                                                                                                 
 975533 root      20   0   20.8g  15.5g  11288 S   0.0   2.1   0:00.00 worker                                                                                                                                                                                                 
 975534 root      20   0   20.8g  15.5g  11288 S   0.0   2.1   0:00.00 worker

The workaround was to disable nx_huge_pages_recovery_ratio.

echo 0 > /sys/module/kvm/parameters/nx_huge_pages_recovery_ratio

We have not seen any issues since implementing the workaround. The fix is to roll back to kernel 6.17.x, but we have not tested this. It does not look like this bug has been logged upstream yet.

Analysis of the kernel code by Claude points at this as a bug in the kernel introduced in 6.19+.

The defect: kvm_recover_nx_huge_pages() sizes its loop from a per-MMU-type counter, kvm->arch.possible_nx_huge_pages[mmu_type].nr_pages. The unaccount path does not honour the type:

static void unaccount_nx_huge_page(struct kvm *kvm, struct kvm_mmu_page *sp)
{
sp->nx_huge_page_disallowed = false;
untrack_possible_nx_huge_page(kvm, sp, KVM_SHADOW_MMU); /* hardcoded */
}

It takes no mmu_type and always passes KVM_SHADOW_MMU. So on a TDP MMU guest, which is every guest on current Intel and AMD hardware, recovering a page decrements the shadow counter instead of the TDP one. The corrective decrement in tdp_mmu_unlink_sp() never
runs, because unaccount_nx_huge_page() cleared sp->nx_huge_page_disallowed on the line above and it returns early.

  ┌───────────────────────────┬─────────────────────────────┬───────────┐
  │          counter          │           actual            │ intended  │
  ├───────────────────────────┼─────────────────────────────┼───────────┤
  │ kvm->stat.nx_lpage_splits │ -1                          │ -1        │
  ├───────────────────────────┼─────────────────────────────┼───────────┤
  │ nr_pages[KVM_TDP_MMU]     │ unchanged                   │ -1        │
  ├───────────────────────────┼─────────────────────────────┼───────────┤
  │ nr_pages[KVM_SHADOW_MMU]  │ -1 (underflows toward 2^64) │ unchanged │
  └───────────────────────────┴─────────────────────────────┴───────────┘

Consequence: nr_pages[KVM_TDP_MMU] is incremented on every split and never decremented, growing monotonically for the life of the guest. The loop bound derived from it grows without limit, so the pass exits only when the list empties, while vCPUs re-fault and re-split pages onto it as it runs. Measured on vm: 305,168 calls to kvm_tdp_mmu_zap_possible_nx_huge_page in 130 s where the design intends 4, a factor of 76,292.

nx_lpage_splits is maintained correctly, so the debugfs gauge looks sane while the real driver is a hidden counter not exposed anywhere. Cost therefore runs inverse to the visible split count and tracks guest age instead. The worker is a vhost_task inside the QEMU thread group, so its time bills to the guest as proxmox.qemu.cpu, and it is bursty (a few seconds at 99% of a core, then ~50 s asleep), which cumulative sampling misses entirely.

Fix: roll back to proxmox-kernel-6.17 (6.17.13-21, pve-enterprise), which predates the per-MMU-type array and has mitigation parity with 7.0.14. Needs a maintenance window.

 

[j.theisen]

Hi @waynej,

thanks for posting on the forum!

Fix: roll back to proxmox-kernel-6.17 (6.17.13-21, pve-enterprise), which predates the per-MMU-type array and has mitigation parity with 7.0.14. Needs a maintenance window.

For clarification is 7.0.14 working properly or is this version still affected?
Since your topic states 7.0.X i assume this would still be affected but want to ask before i look into it in more detail.

Best regards
Jonas

 

[dw-cj]

windows or linux guests? does guest OS / workload matter?

 

[waynej]

Hi @waynej,

thanks for posting on the forum!

For clarification is 7.0.14 working properly or is this version still affected?
Since your topic states 7.0.X i assume this would still be affected but want to ask before i look into it in more detail.

Best regards
Jonas

Anything after 6.19 is affected. 6.17.13-21 has the same CPU vulnerability mitigations as 7.0.14, as opposed to 6.14.x kernels which are missing some mitigations that 6.17.x and 7.0.x cover.

 

[waynej]

windows or linux guests? does guest OS / workload matter?

It affected Windows and Linux guests.

You can check by monitoring top for a VM with:

top -H -b -p $(cat /var/run/qemu-server/<VMID>.pid)

If you see CPU usage for kvm-nx-lpage-re spike around every minute if it is affected. I wakes up every 60s by default, so you might see some usage, but it shouldn't burn a whole core for a couple of seconds. On our worst affected VMs this would spike to 99%.

 

[woma]

I am on 7.0.14-12.
I checked VMs ( ~ 10 with highest cpu usage of cluster) and all have TIME+ of 00:00:00 for "kvm-nx-lpage-re".

So no "cpu burning" here as it would increase TIME+ like in your console output with 15:43.

 

[waynej]

I am on 7.0.14-12.
I checked VMs ( ~ 10 with highest cpu usage of cluster) and all have TIME+ of 00:00:00 for "kvm-nx-lpage-re".

So no "cpu burning" here as it would increase TIME+ like in your console output with 15:43.

That is possible, AFAIK, it should only affect systems when iTLB multihit (CVE-2018-12207) mitigation is enabled. If the CPU is not vulnerable, the mitigation has been turned off or if nx_huge_pages_recovery_ratio=0 you wouldn't any CPU time for that process.

To check: cat /sys/devices/system/cpu/vulnerabilities/itlb_multihit
If it is KVM: Mitigation: Split huge pages then the mitigation is active. If it is KVM: Vulnerable the vulnerability is present, but mitigation has been turned off.

The VM nx_lpage_splits is also an indicator. Run find /sys/kernel/debug/kvm -name nx_lpage_splits -exec sh -c 'echo -n "$1: "; cat "$1"' _ {} \; If they are not zero, then the mitigation is active.

The mitigation can be disabled entirely with echo N > /sys/module/kvm/parameters/nx_huge_pages . This does leave the system vulnerable. Not recommended if there is untrusted software/workloads or multi-tenant. On single-tenant environments or if the workloads are trusted, it shouldn't be an issue.

 

[woma]

Thanks for clarification.
So basically you should not be affected if your CPU desgin is 2019 or newer as CVE is dated 2018.

My rather new AMD Epyc are not affected.

root@xxxxxx ~ # cat /sys/devices/system/cpu/vulnerabilities/itlb_multihit
Not affected

 

[j.theisen]

Thanks for clarifying at all!

Will try to verify it now on an "old" Xeon E5-2620 v4 which should still be affected.

Best regards
Jonas

 

[j.theisen]

So tried it very quickly for now with the aforementioned E5-2620 v4:

# uname -r
7.0.14-14-pve

# cat /sys/devices/system/cpu/vulnerabilities/itlb_multihit
KVM: Mitigation: Split huge pages

# cat /sys/module/kvm/parameters/nx_huge_pages_recovery_ratio
60

Used a Debian 13 VM with x86_64_v3 CPU type and stress-ng --vm 1 as stressor inside the guest.
Yielded no CPU spikes and no time spent in kvm-nx-lpage-re

   2482 root      20   0   10.0g   1.2g   7892 S   0.0   1.9   0:00.00 kvm-nx-lpage-re

Could you @waynej share a little more details on the used VM configuration or if you already see something i overlooked, please tell me what i should do differently.

Best regards
Jonas

 

[David Herselman]

Magic! This solved an issue that we've also been hunting down since upgrading our Proxmox cluster to v9 back in May of 2026. The following graphs are before we mitigated, in way of comparison, from a VM on the host. Load average dropped from 1 with peaks of up to 5.8, to 0.19 with peaks of up to 1:




After mitigation:



Temporary mitigation involves simply live migrating VMs, which re-consolidates memory pages. The issue appears to relate to software mitigation for iTLB Multihit (specifically CVE-2018-12207). The host runs virtual firewalls, where these are less susceptible to iTLB Multihit due to only us having administrative access, we were subsequently able to validate that they remain performant by disabling this mitigation:

echo N > /sys/module/kvm/parameters/nx_huge_pages

 

[David Herselman]

Could you @waynej share a little more details on the used VM configuration or if you already see something i overlooked, please tell me what i should do differently.

The symptom is particularly evident on virtual CheckPoint virtual machines, but it can take 24+ hours to develop. We also noticed this on Windows SQL Servers (128 GiB RAM VM) where we do thick provisioning (we have no swapping on the host and whilst we enable balloon we match the min memory to max memory to avoid over provisioning.

Simply live migrating the VM to another node and back again restores performance.

 

[waynej]

So tried it very quickly for now with the aforementioned E5-2620 v4:

# uname -r
7.0.14-14-pve

# cat /sys/devices/system/cpu/vulnerabilities/itlb_multihit
KVM: Mitigation: Split huge pages

# cat /sys/module/kvm/parameters/nx_huge_pages_recovery_ratio
60

Used a Debian 13 VM with x86_64_v3 CPU type and stress-ng --vm 1 as stressor inside the guest.
Yielded no CPU spikes and no time spent in kvm-nx-lpage-re

   2482 root      20   0   10.0g   1.2g   7892 S   0.0   1.9   0:00.00 kvm-nx-lpage-re

Could you @waynej share a little more details on the used VM configuration or if you already see something i overlooked, please tell me what i should do differently.

Best regards
Jonas

We currently use Cascadelake-Server-v5 as the CPU Type. We experienced the issue using Casacadelake-Server.

Feedback from Claude on replicating the issue:

---

stress-ng --vm 1 won't reproduce this. The iTLB multihit mitigation only splits a huge page when the guest executes from it. --vm does data reads and writes, so it creates zero NX splits no matter how much memory it touches — the split list stays empty and the recovery worker has nothing to do.

The E5-2620 v4 is a fine platform for this (Broadwell is affected by iTLB multihit). Check the preconditions on the host first:

cat /sys/devices/system/cpu/vulnerabilities/itlb_multihit   # want "KVM: Mitigation: Split huge pages"
cat /sys/module/kvm/parameters/nx_huge_pages                # want Y
cat /sys/kernel/mm/transparent_hugepage/enabled             # want [always] or madvise

Give the test VM at least 4 vCPUs — a lot of the symptom is vCPU threads contending with the worker on mmu_lock, and a 1-vCPU guest won't show it.

Watch the actual counter rather than guessing from top:

mount -t debugfs none /sys/kernel/debug
cat /sys/kernel/debug/kvm/*/nx_lpage_splits

Compare that to the same number on an affected production host. If it stays at 0 in the lab, the preconditions are wrong and no workload will help.

To generate splits, you need the guest executing code across many distinct 2 MiB regions. This runs inside the guest (THP set to always):

#!/usr/bin/env python3
import ctypes, time

REGIONS = 4096          # 4096 * 2 MiB = 8 GiB, size to your guest RAM
HUGE = 2 * 1024 * 1024

libc = ctypes.CDLL("libc.so.6", use_errno=True)
libc.mmap.restype = ctypes.c_void_p
libc.mmap.argtypes = [ctypes.c_void_p, ctypes.c_size_t, ctypes.c_int,
ctypes.c_int, ctypes.c_int, ctypes.c_long]
libc.madvise.argtypes = [ctypes.c_void_p, ctypes.c_size_t, ctypes.c_int]

base = libc.mmap(None, (REGIONS + 1) * HUGE, 1|2|4, 0x02|0x20, -1, 0)
if not base or base == ctypes.c_void_p(-1).value:
raise OSError(ctypes.get_errno(), "mmap failed")

aligned = (base + HUGE - 1) & ~(HUGE - 1)
libc.madvise(ctypes.c_void_p(aligned), REGIONS * HUGE, 14)   # MADV_HUGEPAGE

fn_type = ctypes.CFUNCTYPE(None)
funcs = []
for i in range(REGIONS):
addr = aligned + i * HUGE
ctypes.c_ubyte.from_address(addr).value = 0xC3   # RET
funcs.append(fn_type(addr))

print(f"executing across {REGIONS} regions")
while True:
for f in funcs:
f()
time.sleep(0.05)

Each region gets one byte written (faulting in a THP) and one instruction executed inside it. The execution is what forces KVM to shatter the 2 MiB EPT entry into 4K. nx_lpage_splits should climb immediately.

If you want the pathological state in minutes rather than days, amplify the worker on the test host while that runs:

echo 1   > /sys/module/kvm/parameters/nx_huge_pages_recovery_ratio
echo 100 > /sys/module/kvm/parameters/nx_huge_pages_recovery_period_ms

That's not the natural bug, but it puts the thread in the same state and gives you somewhere safe to test mitigations before touching production.

Note: a small test box won't reproduce the scaling behaviour of a large host with many VMs. Use it to confirm the mechanism and test fixes, not to decide whether production is affected.

 

[waynej]

So tried it very quickly for now with the aforementioned E5-2620 v4:

# uname -r
7.0.14-14-pve

# cat /sys/devices/system/cpu/vulnerabilities/itlb_multihit
KVM: Mitigation: Split huge pages

# cat /sys/module/kvm/parameters/nx_huge_pages_recovery_ratio
60

Used a Debian 13 VM with x86_64_v3 CPU type and stress-ng --vm 1 as stressor inside the guest.
Yielded no CPU spikes and no time spent in kvm-nx-lpage-re

   2482 root      20   0   10.0g   1.2g   7892 S   0.0   1.9   0:00.00 kvm-nx-lpage-re

Could you @waynej share a little more details on the used VM configuration or if you already see something i overlooked, please tell me what i should do differently.

Best regards
Jonas

Regarding my earlier response. I tested this on another system and it was not reporting the numbers expected. The python script forces NX splits and demonstrates that splitting happens. However, the bug is driven by cumulative split/recovery events across the guests life and could take hours of continuous churn to build counters worth looking at.

I explored further with Claude and a better way to validate is to check counters with bpftrace. This does not need a workload or stress tool, any VM with a few days of uptime shows it. The python script can still be used to aggressively increment the counters.

apt install bpftrace

bpftrace -e '
  kprobe:kvm_arch_vcpu_ioctl_run
  {
    $kvm = ((struct kvm_vcpu *)arg0)->kvm;
    $pid = $kvm->userspace_pid;
    @splits[$pid]     = $kvm->stat.nx_lpage_splits;
    @nr_pages_0[$pid] = $kvm->arch.possible_nx_huge_pages[0].nr_pages;
    @nr_pages_1[$pid] = $kvm->arch.possible_nx_huge_pages[1].nr_pages;
  }
  interval:s:20 { exit(); }'

Index 0 is KVM_SHADOW_MMU, index 1 is KVM_TDP_MMU (enum kvm_mmu_type,
arch/x86/include/asm/kvm_host.h).

Keys are QEMU PIDs; map them to VMIDs with:

for f in /run/qemu-server/*.pid; do
    echo "vmid=$(basename $f .pid) pid=$(cat $f)"
  done

The probe fires on vCPU entry, so a completely idle guest may not report.
Touch it (a keypress on the console is enough) or extend the interval.

This is my output from a host running 7.0.6-2-pve:

@nr_pages_0[5001]: 18446744073705765164
  @nr_pages_0[2277980]: 18446744073707396970
  @nr_pages_1[2277980]: 2158317
  @nr_pages_1[5001]: 3787683
  @splits[5001]: 1231
  @splits[2277980]: 3671

Signature A — the shadow counter has underflowed.

nr_pages is u64, so subtract from 2^64 = 18446744073709551616:

PID 5001     18446744073705765164  ->  -3,786,452
PID 2277980  18446744073707396970  ->  -2,154,646

Neither guest uses shadow paging. That counter should never have been
incremented, so it cannot legitimately be negative. Anything in the
1.8e19 range here is the defect.

Signature B — the residual equals nx_lpage_splits exactly.

3,787,683 + (-3,786,452) = 1,231   (splits = 1,231)
2,158,317 + (-2,154,646) = 3,671   (splits = 3,671)

If the TDP counter is incremented on every split and never decremented,
the shadow counter is decremented on every recovery and never
incremented, and nx_lpage_splits is maintained correctly on both paths,
then:

nr_pages[TDP] + nr_pages[SHADOW] = splits - recoveries = nx_lpage_splits

It holding to the unit on two guests of different age and workload is not coincidence.

Signature C — how far the loop bound is out.

to_zap = DIV_ROUND_UP(nr_pages[TDP], ratio). With ratio=60 the intended
value is DIV_ROUND_UP(nx_lpage_splits, 60):

PID       actual to_zap   intended   inflation
5001              63,129         21     3,006x
2277980           35,972         62       580x