Nested ESXi - vm's don't start

[bobmc]

I'm trying to run ESXi 7.0.3 under proxmox v8.0.4 on an AMD Ryzen host

ESXi installs OK but I can't get a VM to start. I think the PVE Host is configured ok as I am able to run a VM on a nested PVE instance.

ESXi VM config
balloon: 0 boot: order=sata0;ide2;net0 cores: 6 cpu: host ide2: ProxLib:iso/VMware-VMvisor-Installer-7.0U3n-21930508.x86_64.iso,media=cdrom,size=391216K memory: 8192 meta: creation-qemu=8.0.2,ctime=1696158590 net0: vmxnet3=BE:99:8A:14:E1:34,bridge=vmbr0 numa: 0 ostype: l26 sata0: VMS:vm-1001-disk-0,size=32G sata1: VMS:vm-1001-disk-1,size=100G,ssd=1 smbios1: uuid=4a64aa47-2774-42eb-97c0-5a797c44f33a sockets: 1 vmgenid: a96dd967-7682-4a9c-8e83-ece6da0b286a

There is no error message in the GUI, the VM just fails to start. The tail of the VMKernel warning logs are below

2023-10-01T11:47:48.373Z cpu3:134897)WARNING: UserObj: 5569: vsi_traverse: Unimplemented operation on 0x4540c57e2fa8/SOCKET_VMCI 2023-10-01T11:47:50.900Z cpu3:134927)WARNING: UserObj: 5569: vsi_traverse: Unimplemented operation on 0x4540c57e2fa8/SOCKET_VMCI 2023-10-01T11:47:57.413Z cpu5:134897)WARNING: MemTier: 520: MemTierSummary is not initialized 2023-10-01T11:47:57.413Z cpu5:134897)WARNING: MemTier: 402: MemTierSummary is not initialized 2023-10-01T11:47:57.414Z cpu5:134897)WARNING: cswitch: VLAN_PortGetVLANData:463: [nsx@6876 comp="nsx-esx" subcomp="vswitch"]No vlan data for non dvs ports or ports without port group 0x4000003 2023-10-01T11:47:57.414Z cpu5:134897)WARNING: cswitch: VLAN_PortGetVLANData:463: [nsx@6876 comp="nsx-esx" subcomp="vswitch"]No vlan data for non dvs ports or ports without port group 0x84000004 2023-10-01T11:47:57.414Z cpu5:134897)WARNING: cswitch: VLAN_PortGetVLANData:463: [nsx@6876 comp="nsx-esx" subcomp="vswitch"]No vlan data for non dvs ports or ports without port group 0x4000005 2023-10-01T11:48:00.338Z cpu3:134927)WARNING: MemTier: 520: MemTierSummary is not initialized 2023-10-01T11:48:00.338Z cpu3:134927)WARNING: MemTier: 402: MemTierSummary is not initialized 2023-10-01T11:48:00.338Z cpu3:134927)WARNING: cswitch: VLAN_PortGetVLANData:463: [nsx@6876 comp="nsx-esx" subcomp="vswitch"]No vlan data for non dvs ports or ports without port group 0x4000003 2023-10-01T11:48:00.338Z cpu3:134927)WARNING: cswitch: VLAN_PortGetVLANData:463: [nsx@6876 comp="nsx-esx" subcomp="vswitch"]No vlan data for non dvs ports or ports without port group 0x84000004 2023-10-01T11:48:00.338Z cpu3:134927)WARNING: cswitch: VLAN_PortGetVLANData:463: [nsx@6876 comp="nsx-esx" subcomp="vswitch"]No vlan data for non dvs ports or ports without port group 0x4000005 2023-10-01T11:48:14.688Z cpu3:135251)WARNING: UserObj: 5569: vmkvsitools: Unimplemented operation on 0x4540c57e2fa8/SOCKET_VMCI 2023-10-01T11:48:20.165Z cpu1:131096)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "t10.ATA_____QEMU_DVD2DROM____________________________QM00003_____________" state in doubt; requested fast path state update... 2023-10-01T11:48:33.125Z cpu1:135507)WARNING: UserFile: 2298: rm: Directory changing too often to perform readdir operation (11 retries), returning busy 2023-10-01T11:48:33.129Z cpu1:135507)WARNING: UserFile: 2298: rm: Directory changing too often to perform readdir operation (11 retries), returning busy 2023-10-01T11:48:33.133Z cpu1:135507)WARNING: UserFile: 2298: rm: Directory changing too often to perform readdir operation (11 retries), returning busy 2023-10-01T11:48:33.137Z cpu1:135507)WARNING: UserFile: 2298: rm: Directory changing too often to perform readdir operation (11 retries), returning busy 2023-10-01T11:48:33.143Z cpu1:135507)WARNING: UserFile: 2298: rm: Directory changing too often to perform readdir operation (11 retries), returning busy 2023-10-01T12:20:29.667Z cpu4:131098)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "t10.ATA_____QEMU_DVD2DROM____________________________QM00003_____________" state in doubt; requested fast path state update... 2023-10-01T12:20:29.691Z cpu1:136568)WARNING: World: vm 136568: 8442: vmm0:t1:vcpu-0:Invalid VMCB.

Does anyone know if there's a way to run ESXi nested on Proxmox? Would really be useful for my work currently - thanks

 

[itobin]

vmm0:t1:vcpu-0:Invalid VMCB

Yep, this is a bug and exists in kernel 6, its still not fixed.

Only work around is to boot kernel 5. Its a pain but has been around for a few months.

 

[spirit]

seem to come from this commit
https://git.kernel.org/pub/scm/linu.../?id=174a921b6975ef959dd82ee9e8844067a62e3ec1

according this thread https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2008583


(Maybe open a bugzilla, and ask to proxmox team to revert this commit from kernel. It's pretty small anyway. not sure about the impact)

 

[itobin]

to be honest im suprised no one else has reported it on the forums

 

[bobmc]

I can confirm running kernel 5.13.19-6-pve resolves the issue and allows nested ESXI to run vm's

 

[Neobin]

For reference:
https://bugzilla.proxmox.com/show_bug.cgi?id=4985

 

[itobin]

this is now fixed under kernel 6.2.16-19