Nested PVE (on PVE host) Kernel panic Host injected async #PF in kernel mode

Seeing this issue with nested PVE on PVE on EPYC 9124 CPU. Host has SWAP and KSM enabled, although it has plenty of free memory (100GB of 512GB). This cluster currently runs PVE8.4.14 and runs other workloads too besides nested PVE VMs, with Debian, Ubuntu and Windows Server guest OS. Only the nested PVE VMs are affected by the kernel panic mentioned in this thread.
 
I will update both host and guest to latest pve kernel and test when a window becomes available and will report here after.
Let's hope it was fixed at some point...
 
Sadly it failed again...

Again another Async Page Fault

Host: pve-manager/9.2.3/d0fde103346cf89a (running kernel: 7.0.6-2-pve)
Guest: pve-manager/9.2.3/d0fde103346cf89a (running kernel: 7.0.6-2-pve)

Panic logs of the guest here:
Code: 
[153361.974474] Kernel panic - not syncing: Host injected async #PF in kernel mode
[153361.978993] CPU: 4 UID: 0 PID: 394648 Comm: cut Tainted: P           O        7.0.6-2-pve #1 PREEMPT(lazy)
[153361.982973] Tainted: [P]=PROPRIETARY_MODULE, [O]=OOT_MODULE
[153361.985479] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.17.0-0-gb52ca86e094d-prebuilt.qemu.org 04/01/2014
[153361.989287] Call Trace:
[153361.990120]  <TASK>
[153361.990880]  dump_stack_lvl+0x5f/0x90
[153361.994809]  dump_stack+0x10/0x18
[153361.996472]  vpanic+0x227/0x460
[153361.998465]  panic+0x67/0x70
[153361.999692]  ? early_xen_iret_patch+0xc/0xc
[153362.001405]  __kvm_handle_async_pf+0xa6/0xb0
[153362.004612]  exc_page_fault+0xc4/0x1c0
[153362.007194]  asm_exc_page_fault+0x27/0x30
[153362.009383] RIP: 0010:rep_stos_alternative+0x40/0x80
[153362.011983] Code: c9 75 f6 c3 cc cc cc cc 48 89 07 48 83 c7 08 83 e9 08 74 ef 83 f9 08 73 ef eb de 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 <48> 89 07 48 89 47 08 48 89 47 10 48 89 47 18 48 89 47 20 48 89 47
[153362.018863] RSP: 0018:ffffd4124d003b80 EFLAGS: 00050202
[153362.021286] RAX: 0000000000000000 RBX: 00005d9ccae750a0 RCX: 0000000000000f60
[153362.023799] RDX: 00005d9ccae752c0 RSI: 0000000000000000 RDI: 00005d9ccae750a0
[153362.026384] RBP: ffffd4124d003bc0 R08: 0000000000000000 R09: 0000000000000000
[153362.029068] R10: 0000000000000000 R11: 0000000000000000 R12: ffff8c4a1c0e8918
[153362.031643] R13: 0000000000000003 R14: 00005d9ccae74000 R15: 0000000000000000
[153362.034185]  ? elf_load+0x243/0x270
[153362.037219]  load_elf_binary+0x6b5/0x1930
[153362.039304]  ? load_misc_binary+0x22f/0x340 [binfmt_misc]
[153362.041719]  bprm_execve+0x2d1/0x590
[153362.043742]  do_execveat_common.isra.0+0x18d/0x1a0
[153362.045208]  ? do_getname+0x52/0x1e0
[153362.046602]  __x64_sys_execve+0x3e/0x70
[153362.048074]  x64_sys_call+0xc63/0x2390
[153362.050738]  do_syscall_64+0x10b/0x14e0
[153362.053223]  ? count_memcg_events+0xeb/0x190
[153362.056182]  ? handle_mm_fault+0x254/0x370
[153362.058782]  ? do_user_addr_fault+0x2f8/0x820
[153362.060927]  ? irqentry_exit+0xb2/0x600
[153362.062609]  ? clear_bhb_loop+0x50/0xa0
[153362.065034]  entry_SYSCALL_64_after_hwframe+0x76/0x7e
[153362.067099] RIP: 0033:0x705059bf5e17
[153362.068442] Code: Unable to access opcode bytes at 0x705059bf5ded.
[153362.071103] RSP: 002b:00007fff3a2179c8 EFLAGS: 00000202 ORIG_RAX: 000000000000003b
[153362.075405] RAX: ffffffffffffffda RBX: 000058bd1b6d57a8 RCX: 0000705059bf5e17
[153362.082192] RDX: 000058bd1b6d5728 RSI: 000058bd1b6d5708 RDI: 000058bd1b6d57a8
[153362.084954] RBP: 000058bd1b6d5708 R08: 00007fff3a219e51 R09: 0000000000000001
[153362.087429] R10: 000058bcfc2f2987 R11: 0000000000000202 R12: 000058bcfc2e8102
[153362.089892] R13: 00007fff3a217ab8 R14: 0000000000000002 R15: 000058bd1b6d5728
[153362.092356]  </TASK>
[153362.094022] Kernel Offset: 0x32400000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff)
[153362.101907] ---[ end Kernel panic - not syncing: Host injected async #PF in kernel mode ]---
 
Update:

I've tried adding no-kvmapf to the kernel arguments, it is supposed to disable the Page Fault system altogether.
https://docs.kernel.org/admin-guide/kernel-parameters.html

Guest has been up for 6 days and so far no problems, but I guess this behaviour is non- ideal, as i understand the guest PVE blocks the cpu thread on the PF till that memory chunk is available.

the whole cmdline is as follows:
/boot/vmlinuz-7.0.14-6-pve root=/dev/mapper/pve-root ro no-kvmapf console=ttyS0,115200 earlyprintk=ttyS0,115200 quiet

I found no way to disable kvm-asyncpf while maintaining kvm-asyncpf-int. As supposeldly the latter is the newer supported one and the former the troublesome one.
Any clues on how to expose the host cpu, with kvm-asyncpf-int but no kvm-asyncpf-int?

Bash: 
[   41.139569] clocksource: Long readout interval, skipping watchdog check: cs_nsec: 2242683589 wd_nsec: 2242682958
[   67.898810] tap100i0: entered promiscuous mode
[   67.941924] vmbr20: port 2(tap100i0) entered blocking state
[   67.941928] vmbr20: port 2(tap100i0) entered disabled state
[   67.941940] tap100i0: entered allmulticast mode
[   67.942010] vmbr20: port 2(tap100i0) entered blocking state
[   67.942012] vmbr20: port 2(tap100i0) entered forwarding state
[164464.355866] hrtimer: interrupt took 7194691 ns
 
VictorSTS said:
In my case, it works correctly as soon as disk swap is disabled in the host (i.e. works fine with zram or with no swap, haven't tested zswap yet).
Click to expand...
yeah, that is another half "fix",
'cause, I think, no PF can happen if no swap is available. So we are avoiding the problem instead of fixing it.

But nevertheless that goes against Proxmox own guidelines and could be detrimental for non-nested workflows on the host.

In my use-case, I do need to have swap enabled on the host, cause a lot of the VMs and LXC have used memory that not frequently accessed and swapping it out improves overall memory availability, which can be used for IO cache and other useful stuff.


BTW
@l.leahu-vladucu were you (or any colleagues) able to replicate the faults at some point? Have anyone find any leads on how to tackle the root problem?
 
Darkangeel_hd said:
yeah, that is another half "fix",
'cause, I think, no PF can happen if no swap is available. So we are avoiding the problem instead of fixing it.
Click to expand...
Disabling disk swap is enough. PF are still enabled when using zram as swap and the problem has never happened with that enabled, so the problem is related to disk swap on the host but can't really point my finger to a specific component (kernel, QEMU, etc).

For me is a fix in the sense that it works without panics for months and I prefer that to having any amount of extra ram available, although I would really like to see this fixed sooner than later.
 
VictorSTS said:
Disabling disk swap is enough. PF are still enabled when using zram as swap and the problem has never happened with that enabled, so the problem is related to disk swap on the host but can't really point my finger to a specific component (kernel, QEMU, etc).

For me is a fix in the sense that it works without panics for months and I prefer that to having any amount of extra ram available, although I would really like to see this fixed sooner than later.
Click to expand...
We kinda know why the problem happens, more or less...

if the VM tries to access a memory zone that the host has currently swapped a PF will be generated so that the guest kernel can do something else on that vCPU while waiting, this behavior is normal.
There are two mechanism by which the host can talk this PF to the guest kernel
- the older one implemented with the kvm-asyncpf qemu flag, that uses cpu exceptions for both "Page Not Present" and "Page Ready" signaling
if that "Page Ready" exception is send while the guest kernel cannot handle it, that guest kernel will panic
- the newer 5.10+, implemented with the kvm-asyncpf-int qemu flag, this one uses the normal exception for the PF, but handles the "Page Ready" signal through an async interruption, which can wait till the kernel can process it.

the panics on the guest PVE kernel seem to happen cause the exceptions, be it the Page nor Found or the Page Ready are being sent while the kernel is doing kernel level stuff that cannot be disrupted.

The thing is in not sure how to disable kvm-asyncpf while keeping kvm-asyncpf-int or if that is possible at all.

In case that the panics are happening because of the "Page Not Present" exceptions, then not much can be done other than either disabling the whole PF mechanism or disabling swap, each with its own implications,
if swap is disabled, we have to deal with no swap, which is non-ideal
if the PF mechanism is disabled then the vCPU threads will freeze for some milliseconds till the memory is brought back to ram and they are thawed by qemu/kvm, which can also be non-ideal for certain workflows.

I'm not sure if just using zram for swap exhibits a different behavior than a disk based one.
So it may just be that either the kernel handles swapping in the pages differently or that with a smaller size swap the PF are statistically more rare.

So we are put in a "choose your poison" situation.
 
You must log in or register to reply here.
Top Bottom
Back