Virtualization based security in Windows guests

[dorseymet]

Good morning,
I'm trying to get VBS working in Windows guests. This feature is working on VMware vSphere hosts, but I'm running into some issues with Proxmox. I'm able to enable it in Windows guest if I configure it with vIOMMU option set to VirtIO. However that leads to really high CPU usage, and unrecognized PCI device showing up in device manager, with no drivers available for it. Hardware id PCI\VEN_1AF4&DEV_1057&SUBSYS_11001AF4&REV_01. Vendor id comes up as Red Hat, but device id is a mystery.

If vIOMMU is set to intel, mysterious device disappears, CPU usage goes back to normal, but none of the virtualization security features, can be enabled. Going into Device Security -> Core Isolation, and enabling Memory Integrity prompts for reboot, but after reboot Memory Integrity is back in off state. All of the features listed below report as disabled:

Hypervisor Code Integrity	OS	Hypervisor Code Integrity: enabled	no	High
Virtual Secure Mode (VSM)	OS	Virtual Secure Mode: available	no	High
Input–output Memory Management Unit (IOMMU) is in use	OS	Input–output Memory Management Unit: in use	no	Moderate
System Management Mode (SMM) Protections	OS	System Management Mode Protections: available	no	High
Credential Guard	OS	Credential Guard: enabled	no	Moderate
Secure Kernel	OS	Secure Kernel: running	no	High
Memory Access Protection	OS	Memory Access Protection: enabled	no	Moderate
Mode based execution control (MBEC)	OS	Mode-based execution control: available	no	Low
Memory Overwrite Request Control	OS	Memory Overwrite Request Control: enabled	no	Low
Hypervisor Code Integrity (Strict Mode)	OS	Hypervisor Code Integrity (Strict Mode): enabled	no	Low

I'm guessing high CPU usage might be due to missing driver, I don't think nested Virtualization is the problem here, but perhaps I'm wrong. On Esxi these security features are working just fine without much performance impact. This and Veeam are currently the only issues holding us back from ditching VMware.

Any ideas how to solve this? Or how to get it working with vIOMMU intel option?

Hardware in question is Dell PowerEdge r730xd, with Inter Xeon E5-2680 v4, running latest Proxmox. Esxi running on Dell PowerEdge r720xd with Intel Xeon E5-2670, and production servers are running Esxi on Dell PowerEdge r7515, AMD EPYC 7313P CPUs. Perhaps I might have more luck on production servers, but we're not ready to commit until we get these issues resolved.

 

[itNGO]

Did you test with vIOMMU -> VirtIO?
For Intel (AMD Compatible) you need to add the Kernel-Parameters.

 

[dorseymet]

Did you test with vIOMMU -> VirtIO?
For Intel (AMD Compatible) you need to add the Kernel-Parameters.

Yes, with VirtIO option CPU usage spikes and that unknown PCI device shows up in device manager. What kernel parameters do I need to add with Intel option and how would you do that with Windows guests?

 

[itNGO]

Yes, with VirtIO option CPU usage spikes and that unknown PCI device shows up in device manager. What kernel parameters do I need to add with Intel option and how would you do that with Windows guests?

from: https://forum.proxmox.com/threads/where-to-add-intel_iommu-on-and-iommu-pt.131592/post-578296​

Enabling IOMMU​

• Access the Proxmox VE console via an external monitor or through the Shell on the web management interface
• Type and enter: nano /etc/default/grub
• Add intel_iommu=on to GRUB_CMDLINE_LINUX_DEFAULT=”quiet” (See the screenshot below)
• Write Out the settings and Exit
• Run the command update-grub to finalize changes
• Reboot your Vault

 

[dorseymet]

from: https://forum.proxmox.com/threads/where-to-add-intel_iommu-on-and-iommu-pt.131592/post-578296​

Enabling IOMMU​

• Access the Proxmox VE console via an external monitor or through the Shell on the web management interface
• Type and enter: nano /etc/default/grub
• Add intel_iommu=on to GRUB_CMDLINE_LINUX_DEFAULT=”quiet” (See the screenshot below)
• Write Out the settings and Exit
• Run the command update-grub to finalize changes
• Reboot your Vault

These instructions are for the host pve and I have that setup already. Shouldn’t that be setup on Windows guest (and how?), since VBS is basically nested Hyper-V? So you need to expose IOMMU from Hyper-V to Windows guest.

 

[dorseymet]

"The hypervisor is not protecting DMA because IOMMU is not present or not enabled in BIOS"

I get this message with vIOMMU set to VirtIO. My money is on that missing driver again.

 

[RVK]

Hello dorseymet,
on a testsetup i see the same issue: viommu set to virtio and the unknown device (VEN_1AF4EV_1057) shows up in the Windows device manager. In my case however i do not see the high CPU usage.
Did you have any success in resolving this issue i.e. finding the appropriate driver ? Or can we simply disable this device in Windows: what happens then at OS level ?

 

[dorseymet]

Hello dorseymet,
Did you have any success in resolving this issue i.e. finding the appropriate driver ?

No.

Or can we simply disable this device in Windows: what happens then at OS level ?

Nothing good. I had Windows servers stuck at boot with spinning circle. But I'm pushing VBS with group policy, if you set it manually, my guess would be Windows disables VBS.

 

[chriscom]

Are there any news? I have the same issue on my system (Asus W680 ACE with Intel Core i5-14600)

Thanks!

 

[dorseymet]

Are there any news? I have the same issue on my system (Asus W680 ACE with Intel Core i5-14600)

Thanks!

Nope. Unless RedHat develops the driver for that mystery device, I don’t see how this is going to move forward. Perhaps someone that has support contract with RedHat could pressure them for solution.

 

[wbedard]

@dorseymet I was led to your thread by Google while researching the same issue in my homelab. I did finally get it VBS working (i.e. both Credential Guard and HVCI while using the Intel vIOMMU...) through an unexpected side-effect of running Microsoft's CG-readiness PowerShell script. Let me know if you are still experiencing the issue in your OP and I will help walk you through the process that got everything working for me.