Compile custom kernel to fix AMD issue with nested ESXi VMs

[tchuyev]

Hi,

Since kernel 5.16, it is no longer possible to run VMs on top of nested ESXi servers (7.0 or 8.0) on an AMD platform.
Apparently, this bug has been fixed but is still not applied in latest kernel e.g. pve-kernel-6.2.16-3-pve

https://git.kernel.org/pub/scm/linu.../?id=174a921b6975ef959dd82ee9e8844067a62e3ec1

Can someone guide me on how to build my own Linux custom kernel on PVE 8.0 (Debian 12)?
I've Googled and found way too many documents on the subject... and I really don't know where to start.

In a nutshell:
1. What packages/dependencies should I add to PVE to be able to build a custom kernel?
2. How to apply fix 174a921b6975ef959dd82ee9e8844067a62e3ec1 during compilation?

I could obviously pin 5.15 Proxmox Kernel Image at boot but the idea is to benefit from kernel 6.2+ features.

Any help would be greatly appreciated.
Thank you.

 

[spirit]

seem to be include in 6.2 kernel

https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/log/?h=linux-6.2.y&qt=grep&q=nSVM

 

[tchuyev]

It must be a different fix despite kind of similar I'm afraid.
Nested VMs won't boot with kernel above 5.15 on AMD.
Easy to reproduce.

# cat /sys/module/kvm_amd/parameters/nested
1

# apt install bpftrace
# bpftrace -e 'kprobe:__nested_vmcb_check_controls { printf("tlb_ctl: %d\n", *((uint8 *)arg1+60) )}'

Results show issue is still present.

Any chance to get some help on how to build my own custom kernel with the fix?
Thank you so much

 

[fiona]

Hi,
the commit 174a921b6975ef959dd82ee9e8844067a62e3ec1 is included in all recent Proxmox VE (opt-in) kernels. Why do assume it is the fix for your issue?

febner@pve7-dev ..on/pve-kernel/submodules/ubuntu-kernel % git tag --contains 174a921b6975ef959dd82ee9e8844067a62e3ec1
Proxmox-6.1.10-1
Proxmox-6.1.14-1
Proxmox-6.1.15-1
Proxmox-6.2.11-1
Proxmox-6.2.9-1
Ubuntu-5.19.0-24.25
Ubuntu-5.19.0-31.32
Ubuntu-6.1.0-1.1
Ubuntu-6.1.0-12.12
Ubuntu-6.1.0-13.13
Ubuntu-6.1.0-14.14
Ubuntu-6.2.0-16.16
Ubuntu-6.2.0-17.17
Ubuntu-6.2.0-19.19
v5.19
v6.1

 

[tchuyev]

Issue is finally fixed in Proxmox 8.1.x (kernel 6.5.x): VMs can now be created/ran on top of nested ESXi 8.x hypervisors.
Working using PVE No-Subscription repository.
That's great news

 

[fiona]

Glad to hear! For reference, the fix most likely was this commit: https://git.proxmox.com/?p=pve-kernel.git;a=commit;h=3202de9857143bb619f91aa5fae28f950db09647 which is part of Proxmox kernels >= 6.2.16-19