Proxmox 8 Breaks Nested Virtualization

[sterzy]

Hello friend, a question, this patch will be to up for general linux kernel or only is a fix for proxmox kernel?

Our “fix” is a back port of these patches [1] provided by Sean Christopherson on the KVM/Linux kernel mailing list. We don't have any say over what ultimately ends up in the kernel, but if these patches are accepted, then yes future mainline kernel releases will include them too. I'd recommend you follow the upstream discussion on the mailing list to see if and when the patches are applied there.

[1]: https://lore.kernel.org/all/20231018194104.1896415-1-seanjc@google.com/

 

[mario156090]

Our “fix” is a back port of these patches [1] provided by Sean Christopherson on the KVM/Linux kernel mailing list. We don't have any say over what ultimately ends up in the kernel, but if these patches are accepted, then yes future mainline kernel releases will include them too. I'd recommend you follow the upstream discussion on the mailing list to see if and when the patches are applied there.

[1]: https://lore.kernel.org/all/20231018194104.1896415-1-seanjc@google.com/

Thank's a lot, I will check this one twice for week.

 

[mario156090]

Hello, well, I manually patch the kernel with the patch from this thread and nested works, but aparently some features from processors does not work.

 

[sterzy]

Hello, well, I manually patch the kernel with the patch from this thread and nested works, but aparently some features from processors does not work.

Well, hard to tell what is wrong here, since we don't know how exactly you patched the kernel and which version you used. Also, what does the VM's config look like? Can you please test it with the proxmox-kernel-6.2.16-19 kernel in the no subscription repo?

 

[mario156090]

Bueno, es difícil decir qué está mal aquí, ya que no sabemos exactamente cómo parcheaste el kernel y qué versión usaste. Además, ¿cómo es la configuración de la VM? ¿Puedes probarlo con el kernel proxmox-kernel-6.2.16-19 en el repositorio sin suscripción?

Hello, I attach my dmesg output. This weekend I will to install proxmox in my laptop for test with that kernel and try the same escenary. Do you need some log for give you after my test?

 

[deepcloud]

When is the proxmox-kernel-6.2.16.-19 coming to the enterprise repo, we are having a paid subscription to all hosts so don't want to try the pve-no-subscription unless this is going to take time to come to the enterprise repo.

 

[t.lamprecht]

When is the proxmox-kernel-6.2.16.-19 coming to the enterprise repo, we are having a paid subscription to all hosts so don't want to try the pve-no-subscription unless this is going to take time to come to the enterprise repo.

That kernel has been on no-subscription for about two weeks without any known new regression, so I'd say that we'll move that a long to the enterprise repository at the beginning of next week, if nothing new comes up.

 

[t.lamprecht]

FYI that kernel got now moved to the enterprise repository.

 

[deepcloud]

Nested ESXi Works l ike a charm Now on the enterprise repo... Thanks folks

 

[tchuyev]

Issue is finally fixed in Proxmox 8.1.x (kernel 6.5.x): VMs can now be created/ran on top of nested ESXi 8.x hypervisors.
Working using PVE No-Subscription repository.
That's great news

 

[the-last-englishman]

If your doing nested vSAN cluster then chick this post: https://forum.proxmox.com/threads/p...i-are-not-seeing-cpu-flag-cpuid-mwait.137143/ workaround is in the linked KB.

 

[Ludicael]

Is there any update with regards to using VMWare Workstation in a Windows guest VM on Proxmox 8?

Using the latest kernel (at time of writing) 6.5.

ESXi works, I'm able to run this as a guest - but not Workstation.

 

[mikeyo]

Now that kernel 6.8 resolves the amd nested esxi issue, can someone confirm if virtualised iommu is possible so that a gpu or other device can be selected for passthrough on the l2 esxi host? I intend to run proxmox on my x570 3950x, install esxi 8 as a vm and passthrough one of my tesla gpus to a VM running inside esxi.

 

[70tas]

I am also seeing an issue with Vmware Workstation on Windows 11 and Windows 8.1. VMW reports that Workstation and Hyper-V are not compatible, but I don't have Hyper-V installed. Just starting to learn Proxmox, because I work in a heavy Windows shop, but I'm kind of stuck in the sand so to speak.
Thank you

Is there any update with regards to using VMWare Workstation in a Windows guest VM on Proxmox 8?

Using the latest kernel (at time of writing) 6.5.

ESXi works, I'm able to run this as a guest - but not Workstation.

 

[70tas]

Well, looks like I'll be falling back to vMware or a mix of vMware and Proxmox. I need nested processing, and although it has been months since it has been broken in Proxmox, there is no official word of a fix. Too bad. Dear Proxmox, how do you expect people to buy a subscription when it is priced higher than the competition, and the product fails in key areas. BTW, while you may be reading this, consider integration with TrueNAS.

 

[t.lamprecht]

Well, looks like I'll be falling back to vMware or a mix of vMware and Proxmox. I need nested processing, and although it has been months since it has been broken in Proxmox, there is no official word of a fix. Too bad. Dear Proxmox, how do you expect people to buy a subscription when it is priced higher than the competition, and the product fails in key areas. BTW, while you may be reading this, consider integration with TrueNAS.

The original issue of this thread has been "fixed" from our side, since a while, as confirmed by some users just in this thread, even though it wasn't a real necessity w.r.t modern virtualization technologies like KVM.
That said, nested virtualization is by far not a key area for most, and the nested virtualization layer works fine in general, this is just some proprietary product having virtual blockades – so I'd recommend asking your VMWare support rep, as their product is not working.
Else, If you want help here for free from the Proxmox community then post at least some actual details, like what you tried, VM configs, host hardware, ...

 

[70tas]

The original issue of this thread has been "fixed" from our side, since a while, as confirmed by some users just in this thread, even though it wasn't a real necessity w.r.t modern virtualization technologies like KVM.
That said, nested virtualization is by far not a key area for most, and the nested virtualization layer works fine in general, this is just some proprietary product having virtual blockades – so I'd recommend asking your VMWare support rep, as their product is not working.
Else, If you want help here for free from the Proxmox community then post at least some actual details, like what you tried, VM configs, host hardware, ...

Llamprecht:
Obviously it was confirmed by some users, see this thread, with the exception of a Windows VM’s. There are even Bugzilla reports confirming this. It seems obvious that Proxmox is not interested in the Home Lab market, from your reaction. The same with the pricing model of the product, as I would be more than happy to pay for a home lab subscription, as long as the only support answers are not such as this.
I have a great respect for the open community, and did not intrude on paid support forums to ask my question, since I don’t yet have a subscription. The question is will I get a subscription for support? Not if the community forum is not open and inclusive.

 

[t.lamprecht]

Obviously it was confirmed by some users, see this thread, with the exception of a Windows VM’s.

The original issue with ESXi refusing by some magic bit they don't even use was fixed, as that bit is now exposed to the VM any other issues are rather different ones – so no, there's no exception for the specific issue of this specific thread. Your problems have very likely a different root cause.

It seems obvious that Proxmox is not interested in the Home Lab market, from your reaction

Yeah, the ten thousand posts of Proxmox staff themselves, plus a huge community on top of that, helping people here does not really make such an observation all too obvious to me..

Anyhow, IMO VMWare Workstation inside Windows VMs is an odd use case for any setup, be it home lab or enterprise, but what's more relevant, it's a few to many layers away from Proxmox VE. There's VMWare that makes the product you got a problem with to ask first, then there's Windows, the platform you want to make the former run on, which also offer support.

The same with the pricing model of the product

The product itself is 100% free, released under the GNU AGPLv3..

as long as the only support answers are not such as this.

Well, you will be asked for details there too, which you did not yet provide and might have made this conversation a bit more productive, but yes, for a paid support you get guaranteed responses for which the quality naturally depends on the support level (basic subscription is basic support) and if the question is covered by the subscription agreement one agrees to when buying said subscription. VMWare workstation is a proprietary project that we have no ties with whatsoever, so enterprise support w.r.t. that will be obviously limited from our side, but as said, you can ask VMWare about their products.

Not if the community forum is not open and inclusive.

Not sure what you're getting at here? Nobody censored you, but nobody here owes you anything either.
While it won't guarantee you an answer in this 100% free of charge forum, providing details asked is surely increasing that chance for getting it more than this route of conversation here.

Ps. The proprietary world is great on checking some bogus details for refusing to run even if they could, so CPU type (that includes some hyperv features that one can enable/disable) and SMBIOS strings that identify the platform could be at "fault" here.
It might also be the case that VMWare Workstation is just not supporting running like this, we did not make it so can't answer that..

 

[70tas]

The original issue with ESXi refusing by some magic bit they don't even use was fixed, as that bit is now exposed to the VM any other issues are rather different ones – so no, there's no exception for the specific issue of this specific thread. Your problems have very likely a different root cause.

Yeah, the ten thousand posts of Proxmox staff themselves, plus a huge community on top of that, helping people here does not really make such an observation all too obvious to me..

Anyhow, IMO VMWare Workstation inside Windows VMs is an odd use case for any setup, be it home lab or enterprise, but what's more relevant, it's a few to many layers away from Proxmox VE. There's VMWare that makes the product you got a problem with to ask first, then there's Windows, the platform you want to make the former run on, which also offer support.

The product itself is 100% free, released under the GNU AGPLv3..

Well, you will be asked for details there too, which you did not yet provide and might have made this conversation a bit more productive, but yes, for a paid support you get guaranteed responses for which the quality naturally depends on the support level (basic subscription is basic support) and if the question is covered by the subscription agreement one agrees to when buying said subscription. VMWare workstation is a proprietary project that we have no ties with whatsoever, so enterprise support w.r.t. that will be obviously limited from our side, but as said, you can ask VMWare about their products.

Not sure what you're getting at here? Nobody censored you, but nobody here owes you anything either.
While it won't guarantee you an answer in this 100% free of charge forum, providing details asked is surely increasing that chance for getting it more than this route of conversation here.

Ps. The proprietary world is great on checking some bogus details for refusing to run even if they could, so CPU type (that includes some hyperv features that one can enable/disable) and SMBIOS strings that identify the platform could be at "fault" here.
It might also be the case that VMWare Workstation is just not supporting running like this, we did not make it so can't answer that..

Well you guys, is there anything any might say on this forum that won't rub you all the wrong way? Did you all wake up on the wrong side of the bed? It isn't to you to judge what a use case may be, or how far away another product is from Proxmox. The point is that there is a documented problem which started with Proxom 8 and broke nested Windows and VMW, and nested ESXi. At some point ESXi was fixed, but not Windows. Go read this thread and look up the bug reports. I am not attacking you, nor Proxmox; I was happy to be leaving the proprietary world to get to Proxmox. I was just trying to get a response from users in the same scenario as me. If you aren't using Windows 11/10 and VMW you should not be responding to this thread.

 

[t.lamprecht]

Well you guys, is there anything any might say on this forum that won't rub you all the wrong way?

Not feeling attacked, but again, third time's a charm and all that, you might want to start posting the asked details...

If you aren't using Windows 11/10 and VMW you should not be responding to this thread.

This thread is about the ESXi issue, as said, you're VMWare Workstation use case is affected by something else (or in addition), as the original authors ESXi issue is clearly fixed.
So open your own thread (or ask Windows/VMWare support, affects their products), locking this one here as it seems just fruitless after two chances to actually post some meaningful details...