IMHO, it would be very useful for everyone if someone could explain how to reproduce the issue (i.e. how to do those ballooning changes), so we can test the patched kernel in our testing labs too. Thanks everyone for their efforts!
According to the upstream patch [1], the freeze occurs when the
mmu_invalidate_seq counter exceeds 2^31 = 2147483648. I use the following bpftrace script to keep track of the counter value per QEMU process ID:
Code:
kprobe:direct_page_fault {
$ctr = ((struct kvm_vcpu*)arg0)->kvm->mmu_invalidate_seq;
@counts[pid] = $ctr;
}
interval:s:2 {
print(@counts);
print("---\n");
}
You need to install
bpftrace, then you can run it with
bpftrace SCRIPT. Note that I only tested this on Bookworm / PVE 8, the script might not work with the
bpftrace version shipped with Bullseye / PVE 7.4.
Now, to reproduce the freeze, it is necessary to get the counter above 2^31. Under normal operations, the counter seems to increase very slowly. KSM
seems to accelerate it a bit, though I haven't fully tested this yet.
One thing that seems to work is excessive ballooning. I set up a PVE VM with 60000 MiB RAM, and used a script to alternate between 4000 MiB and 60000 MiB. If you want to try this yourself, you can use the script below as a starting point, but please note I threw this together quite quickly, so
use it at your own risk (if you do want to try, replace the VMID accordingly). It might also need some modifications to work with Windows VMs. With this script,
mmu_invalidate_seq reached 2^31 within a couple of hours and the VM froze with 100% CPU and 99% ppoll in the strace output. The increase speed seems proportional to the difference of
HIGH and
LOW, so you might be able to reproduce this faster if you assign more memory to the VM and adjust HIGH accordingly.
In my understanding, the
proxmox-kernel-6.2.16-12-pve should fix this reproducer (so, the VM should not freeze even when the counter exceeds 2^31).
If you try this, we'd be happy about any feedback, also to find other factors that increase the speed at which the counter increments -- ballooning is probably not the only one.
Code:
#!/bin/bash
VMID=SET_VM_ID_HERE
HIGH=60000
LOW=4000
WAIT=1
balloon () {
amount=$1
echo balloon to $amount ...
pvesh create /nodes/localhost/qemu/$VMID/monitor -command "balloon $amount"
echo wait until actual=$amount ...
while true;
do
info=$(pvesh create /nodes/localhost/qemu/$VMID/monitor -command "info balloon")
echo $info
if [[ $info == *"actual=$amount"* ]]; then
break
fi
sleep 2
done
}
while true;
do
balloon $HIGH
sleep 1
balloon $LOW
done
[1]
https://lore.kernel.org/all/f023d92.../T/#maa94d78b12c00071029cc0c8ff2785d6d14bce2e
