[TUTORIAL] FROM NOTHING: Physical disk with Windows 11 as VM + GPU pass-through

[jemme]

Hello!

This is my first time starting a thread on here, but I've posted on an already existing thread about my problem. Since I've solved it and now have a running Windows 11 VM that operates as a "normal computer", I thought I would share my process. Keep in mind that I consider myself a noob in the world of Proxmox-ing, VM-ing, and Linux-ing (are those real terms? Idk they're fun and make sense to me). Also, to make things more fun to read, I'm gonna tell it as a story to start with BUT will get serious when talking about related things I did (please don't delete my post Proxmox mods). Sorry in advance for any English errors, I'm a French Canadian.

This is my first time using this software and diving into the beautiful world of "real computer science" stuff. As my roommate (he's important to the story/tutorial I'm about to tell you) says, "every time I ask you what's wrong with the computer and you explain it to me, I tell myself you are a wizard casting weird spells but as long as I can play DDR with my new pad I won't complain". Yes, this whole story starts as a gift to him: playing Dance Dance Revolution on a PC (look up Stepmania if you're curious). He couldn't figure out how to run it on his laptop, which makes sence since it doesn't have good hardware like my gaming PC.

This gaming PC of mine has a GeForce 3060, which is overkill for Stepmania, but he received a Meta VR headset from his dad. He said I could use it if I set up his pad properly. This is a huuuuuge win for me. Plus, my other roommates want to play Baldur's Gate 3 and Sims 2 so it's a win for everyone. Except for me since I'm the one setting all of it up XD, but this is fun and I love learning new things about computers!!!

Alright, enough goofing around, let's get started with it! The goal is to convert your computer into a VM hosting machine without wiping your important data, but just to be safe, please make a back up of your important files.

Note: If you don't care about the Windows 11 to VM process and are interested in the GPU pass-through, skip ahead! Make sure that your VM is a q35 machine.

Note: If I made mistakes explaining stuff, let me know so I can correct it. I wish that this thread to become a good guide for others. This tutorial is for absolute noobs so stuff like advanced networking like VLANs won't be explored.

My hardware

Components	Brands and such
Motherboard	ASUS Prime B460-PLUS ATX
CPU	Intel Core i7-10700F
RAM*	TeamGroup T-Force Vulcan Z DDR4-3200 (2x16GB)
RAM	Corsaire Vengeance RGB Pro DDR4-3200 (2x8GB)
GPU	GeForce RTX 3060 12GB
GPU*	Quadro P400
Storage	TeamGroup MP33 2TB M.2-2280
Storage*	Western Digital Black SN750 1TB M.2-2280
Storage*	(2x) SAMSUNG 870 EVO SATA III SSD 1TB 2.5”
Power Supply	SuperNOVA 650W GA 80+ Gold

*Added for the server, meaning wasn't part of the original build.

Yes, a lot of stuff listed seems weird (mainly the RAM which will probably make everyone panic), but in theory everything should work. The Windows 11 VM will get 16GB of RAM like the original build so it's like if nothing changed (ish). The Western Digital SSD is used for the Proxmox install since I didn't want to wipe the Windows 11 install and will hold all the VMs and LXC containers that I will install in the future. I wanted to get this tutorial's goal first before diving into something else, such as a Jellyfin media library with video transcoding (hence the 2 Samsung SSDs).

Installing Proxmox
Ok so first off, install Proxmox with the method you want. I personally use a Ventoy USB stick (because it is cool and useful). If you don't know what that is, click here, but see it as a multi-ISO holder where you can choose which OS to install on a new/old machine. I am using Proxmox VE 8.3, so keep that in mind. Be warry of what drive you install it on, you don't want to wipe the Windows 11 install. Once that's done, connect to the machine via the URL it tells you to (it's the one you chose while installing). You can also reserve the IP in your router settings, but this is not the point of this tutorial. I recommand watching the start of this video by Kalos for that.

Once you have access to the webGUI, you're gonna follow these steps. I recommend doing this on every fresh install of Proxmox (as recommended by Techno Tim). This section is credited to him.

• Remove enterprise repositories in Updates -> Repositories
  • Since we are noobs (or broke), we'll have to remove these repositories if we want to run updates. Proxmox will complain about this, but we will fix it don't you worry.
• Add the no-subscription repository
  • This will tell Proxmox where/how to update itself. You'll need to access a file from the command line, so use the nano command. Get used to it, this is how Linux work XD. /etc/apt/sources.list.d/pve-no-enterprise.list is the place we want to go, so nano /etc/apt/sources.list.d/pve-no-enterprise.list is the command! From now on, I will only say the file you need to go so you can learn properly . PS: ctrl+X, Y, Enter to exit and save and use right click + paste when copying lines in the Shell environment.

# not for production use
deb http://download.proxmox.com/debian/pve bookworm pve-no-subscription

• Run run apt-get update && apt dist-upgrade to apply those changes

After that, reboot your computer but access the BIOS since we need to activate some stuff there. You need to find these following settings. At this point, I'm sure you already know that virtualization needs to be activated and already checked if your computer supports it. If not... I'm sorry... Maybe you will get lucky and your BIOS will have a search tool (which was my case)

• Enable virtualization (I wrote vir and it showed up)
• Enable VT-d
• Enable IOMMU grouping (I'm not sure if this exists but I think it does)

If you don't have those options, then you're good to go! If you don't, maybe try updating your BIOS. I had to do it and sweated a lot during the process. Since this is something different for everyone, google your motherboard with BIOS update next to the name and check another tutorial (and come back afterwards!).

Making the Windows 11 VM
Now, let's create the VM that's gonna be ran like a "normal computer". You'll have to download the most recent VirtIO ISO for Windows which you can find on the website linked. Then you'll need to head to your local volume on the server (or the place you want). Then, click on Create VM to start the process. Note that you'll have to check Advanced at the bottom of the window to see all the options.

• Name and ID your VM. I used Windows11 and 100 since I didn't have another VM. DO NOT CHECK START ON BOOT. We actually don't care about what we put here since we will be using a physical disk with Windows 11 already installed.
• The ISO image should be the VirtIO. Select Microsoft Windows for the type of the Guest OS. Of my understanding and reasoning, we actually don't really care about this ISO image since we already have Windows 11 installed on a disk. Check Add additional drive for VirtIO drivers and put the VirtIO ISO here too. This will be used later on.
• For the System, we will use a q35 machine since we want to do GPU pass-through. The BIOS should be OVMF (UEFI). Windows 11 likes to have a EFI and TPM storage, so check those boxes and add a volume. You can also activate the QEMU agent but I didn't since Windows could see that it is a VM and some games don't like that (anti-cheat or something, I don't make the rules). The SCSI Controller should be VirtIO SCSI.
• Delete the virtual disk! We're gonna use a real one.
• Set the CPU and Memory to your needs. With my needs, I have 14 cores (out of 16 that I have) and 16GB of RAM.
• Use the bridge created by default by Proxmox with a VirtIO (paravirtualized) Model.
• Confirm your settings and create the VM, but do not start it!

We need to give the disk to the VM, but we need the information of our disk. Head to the Disks tab of your node. Notice all the expensive disk that you have and be proud for a second. Now be serious and look for the right disk that contains the Windows 11 install. It was easy for me since mine is a NVMe of 2TB, but if you are struggeling focus on the disks with partitions. The disk with the BIOS boot partition is most likely the one with your Proxmox install, so don't look at that one! Once you've found the disk that has Windows 11, write down the Model and the Serial number. This is crucial to our operation. I recommend using Notepad.

Let's mount the disk to the VM with a command! Head to the Shell window and insert the following command. Note that you have to replace the ID, Model, Serial number in the command. This will be different for everyone. Remove any """ present in the commands

If you are using a SATA disk, use this command

qm set """ID""" -sata1 /dev/disk/by-id/ata-"""Model"""_"""Serial Number"""

If you are using a NVMe disk (like me), use this command

qm set """ID""" -sata1 /dev/disk/by-id/nvme-"""Model"""_"""Serial Number"""

Here's how my command looks like

qm set 100 -sata1 /dev/disk/by-id/nvme-TEAM_TM8FP6002T_TPBF2410210070700202

Note: If there's a space in your Model name (like me), replace it by an underscore.

If this didn't work, double check the ID. Then, double check the Model. Then, double check the Serial Number. This took me multiple tries (I'm starting to think I'm dislexyc). If it stills doesn't work, ask for help in this thread!

All that's left to do is to change the boot order of the VM. Head to the Options tab of the VM and click on the Boot Order. Check the new disk that you added and put it at the top of the list.

It is now time to start the VM! You can double click on the ID or head to the Console tab of the VM. Hurray! You now have virtualized your machine! You can pat yourself on the shoulder... and realize that somethings wrong. Why is it not showing up on my monitor? Why is it on another computer? Why doesn't my keyboard and mouse work? And what is this lag? Well my friend, that is because your VM is running on a virtual GPU created by Proxmox (which I don't know how its possible, I should look that up one day) and you are accessing it through the network. So there is only one solution to this beautiful problem...

PCIe Pass-through (GPU and others)
That's right, let's spend three days trying to figure it out together! Except that I already did the work for you so you don't need to cry as much as me (I hope so).

Firstly, we need to prepare Proxmox for our magic spells. I'm hoping you read all of the tutorial and saw the things you have to do in the BIOS. If you didn't (no offense taken, I get it), check right before the section "Making the Windows 11 VM". Next step is to head to the Shell tab.

Run nano /etc/default/grub and change the GRUB_CMDLINE_LINUX_DEFAULT to

GRUB_CDMLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt"

If you have a AMD, replace intel by amd. intel_iommu=on tells Proxmox that we want to group our hardware into different groups and iommu=pt tells Proxmox that we want to do some pass-through action. When you saved the modifications, run update-grub.

Note: I'm aware that there's a second way to boot Proxmox that requires you to modify a different file, but I don't know how to do that or check what boot tool (is that the term?) you are using. Please someone explain it to me and I will add it to the tutorial and credit you!

We now need to tell Proxmox to use certain modules that will assist it and make our spell work (we are wizards, remember?). Head to /etc/modules and insert the following

vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd

*** Apparently, vfio_virqfd is not a thing anymore, but adding it didn't create problems (someone confirm this information please!!!)

Anytime you modify this file, you need to run update-initramfs -u -k all. If this gives you an output similar to this, then we'll have to fix this! If not, then skip ahead to the reboot section right after.

update-initramfs: Generating /boot/initrd.img-6.5.11-7-pve
Running hook script 'zz-proxmox-boot'..
Re-executing '/etc/kernel/postinst.d/zz-proxmox-boot' in new private mount namespace..
No /etc/kernel/proxmox-boot-uuids found, skipping ESP sync.
update-initramfs: Generating /boot/initrd.img-6.5.11-4-pve
Running hook script 'zz-proxmox-boot'..
Re-executing '/etc/kernel/postinst.d/zz-proxmox-boot' in new private mount namespace..
No /etc/kernel/proxmox-boot-uuids found, skipping ESP sync.

The fix is credited to this reddit comment. I will put the comment here for ease of use.

Ok, for any one who stumble on this post with this same problem. Here is the solution i have found and it worked for me.
If you are using ext4 file system with EFI so you are using GRUB please try the following and let me know if it works for you.
To check which partition is /boot with vfat format:

:\~# lsblk -o +FSTYPE

To initialize ESP sync first unmount boot partition:

:\~# umount /boot/efi

Then link the vfat partiton with proxmox-boot-tool:

:\~# proxmox-boot-tool init /dev/XXXXXXXX where XXXXXXXX is the name of vfat partiton from lsblk +FSYSTEM

Then:

:\~# mount -a

Then to update modules:

:\~# update-initramfs -u -k all

Reboot

Thank you so much u/Dunadan-F for this fix, it helped me!

It is time to reboot and check if everything was done correctly. Run lsmod | grep vfio to see if the modules work. My output looks like this. The important part is that the modules we added previously show up.

vfio_pci               16384  0
vfio_pci_core          86016  1 vfio_pci
irqbypass              12288  2 vfio_pci_core,kvm
vfio_iommu_type1       49152  0
vfio                   65536  4 vfio_pci_core,vfio_iommu_type1,vfio_pci
iommufd                94208  1 vfio

We are set! It is now time to check which IOMMU group our GPU (and other devices) is. For this, run for d in /sys/kernel/iommu_groups/*/devices/*; do n=${d#*/iommu_groups/*}; n=${n%%/*}; printf 'IOMMU group %s ' "$n"; lspci -nns "${d##*/}"; done. I found this command in this Proxmox Forum thread, so credit to @leesteken for it! This will list all of the devices you have with their associated IOMMU group and some other useful infomation. Here's how my output looks like... In the replies!

 

[jemme]

So there's a limit of 16384 characters, so this is split into two parts! Like promised, here's the output.

IOMMU group 0 00:00.0 Host bridge [0600]: Intel Corporation 10th Gen Core Processor Host Bridge/DRAM Registers [8086:9b43] (rev 05)
IOMMU group 1 00:01.0 PCI bridge [0604]: Intel Corporation 6th-10th Gen Core Processor PCIe Controller (x16) [8086:1901] (rev 05)
IOMMU group 1 01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GA106 [GeForce RTX 3060] [10de:2503] (rev a1)
IOMMU group 1 01:00.1 Audio device [0403]: NVIDIA Corporation GA106 High Definition Audio Controller [10de:228e] (rev a1)
IOMMU group 2 00:14.0 USB controller [0c03]: Intel Corporation Comet Lake PCH-V USB Controller [8086:a3af]
IOMMU group 3 00:16.0 Communication controller [0780]: Intel Corporation Comet Lake PCH-V HECI Controller [8086:a3ba]
IOMMU group 4 00:17.0 SATA controller [0106]: Intel Corporation 400 Series Chipset Family SATA AHCI Controller [8086:a382]
IOMMU group 5 00:1b.0 PCI bridge [0604]: Intel Corporation Device [8086:a3e9] (rev f0)
IOMMU group 5 00:1b.4 PCI bridge [0604]: Intel Corporation Comet Lake PCI Express Root Port #21 [8086:a3eb] (rev f0)
IOMMU group 5 03:00.0 Non-Volatile memory controller [0108]: Sandisk Corp WD Black SN750 / PC SN730 NVMe SSD [15b7:5006]
IOMMU group 6 00:1c.0 PCI bridge [0604]: Intel Corporation Device [8086:a392] (rev f0)
IOMMU group 6 00:1c.3 PCI bridge [0604]: Intel Corporation Device [8086:a393] (rev f0)
IOMMU group 6 00:1c.4 PCI bridge [0604]: Intel Corporation Comet Lake PCI Express Root Port #05 [8086:a394] (rev f0)
IOMMU group 6 04:00.0 PCI bridge [0604]: ASMedia Technology Inc. ASM1083/1085 PCIe to PCI Bridge [1b21:1080] (rev 04)
IOMMU group 6 06:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller [10ec:8168] (rev 15)
IOMMU group 6 07:00.0 VGA compatible controller [0300]: NVIDIA Corporation GP107GL [Quadro P400] [10de:1cb3] (rev a1)
IOMMU group 6 07:00.1 Audio device [0403]: NVIDIA Corporation GP107GL High Definition Audio Controller [10de:0fb9] (rev a1)
IOMMU group 7 00:1d.0 PCI bridge [0604]: Intel Corporation Comet Lake PCI Express Root Port 9 [8086:a398] (rev f0)
IOMMU group 7 08:00.0 Non-Volatile memory controller [0108]: MAXIO Technology (Hangzhou) Ltd. NVMe SSD Controller MAP1202 [1e4b:1202] (rev 01)
IOMMU group 8 00:1f.0 ISA bridge [0601]: Intel Corporation B460 Chipset LPC/eSPI Controller [8086:a3c8]
IOMMU group 8 00:1f.2 Memory controller [0580]: Intel Corporation Cannon Lake PCH Power Management Controller [8086:a3a1]
IOMMU group 8 00:1f.3 Audio device [0403]: Intel Corporation Comet Lake PCH-V cAVS [8086:a3f0]
IOMMU group 8 00:1f.4 SMBus [0c05]: Intel Corporation Comet Lake PCH-V SMBus Host Controller [8086:a3a3]

Notice that the GeForce 3060 is in group 1 alone. This is important, and is the reason why it took me 3 days to fix my problem. You can find the original post here (link). In short, my original problem was that the GeForce was in the same group as the Ethernet port. When I'd boot my VM, my webGUI would tell me that the server was offline (and the VM didn't have Ethernet either). The only solution to this was to put the GPU in a different PCIe slot. As you can see, my Quadro P400 is in the 6th group with the Ethernet port, meaning that I will have the same problem as before if I ever pass it to a VM. The unfortunate part is that you can't choose which piece goes in which group since it is dependant of your motherboard (to my understanding). I got lucky and my USB Controller is alone in its group too, so I can pass it to the VM and use it like a normal USB port (like we will see later).

There is a work around: you can add pcie_acs_override=downstream,multifunction to your grub file. I personally don't want to do that because of the security issues it can bring, but if you wish to do so, you can. Here's a post on reddit about the risks. (link)

Now, find the GPU you want to pass-through and look at the end of the line. Those numbers in square braquets are the ID of the device. Write down the ID of the GPU and the ID of the audio device associated to it (if your GPU has one, mine does). In /etc/modprobe.d/vfio.conf, write options vfio-pci ids=(id of GPU),(id of audio) disable_vga=1. If you don't have an audio ID, remove the comma and what's after. If you want audio, then you'll have to pass it through (I'm not sure how this is achieved).

There's one thing left to do in the Shell, you got this! Head to /etc/modprobe.d/blacklist.conf so we can blacklist the drivers that the GPU use. From the Proxmox wiki, here's the list of drivers.

• AMD GPUs

echo "blacklist amdgpu" >> /etc/modprobe.d/blacklist.conf
echo "blacklist radeon" >> /etc/modprobe.d/blacklist.conf

• NVIDIA GPUs

echo "blacklist nouveau" >> /etc/modprobe.d/blacklist.conf
echo "blacklist nvidia*" >> /etc/modprobe.d/blacklist.conf

• Intel GPUs

echo "blacklist i915" >> /etc/modprobe.d/blacklist.conf

Reboot the node to apply those changes.

Back in the Hardware tab of the VM, click on Add, then PCIe Device. Select the GPU in the Raw Device menu and check Primary GPU, ROM-Bar and PCI-Express (if you don't see all the options, check Advanced). Do this another time, but select the audio device and check ROM-Bar and PCI-Express only. If you want, you can pass an USB controller like me and checking only All Functions. If you don't have that option, you can still click on Add then USB Device and use the USB Port of your mouse and keyboard, or any other USB device! There's also another way to do it like seen on the Proxmox Wiki (link), but I haven't tried it (yet..?).

You are officially done! If everything goes well, the VM will start. You'll have to use a monitor with a cable hooked into the GPU you just passed-through. Your mouse and keyboard should be working like normal, and the lag should be gone! Don't forget to download the VirtIO drives that we added in the VM creation section. After that, you can shutdown your VM and remove them from the Hardware tab safely.

If you go to the Task Manager and check that Windows knows that it's a VM. If anything, I would recommend giving this tutorial a read (link). There's more stuff to do to hide the fact that Windows is in a VM.

Thank you for giving this a read! I appreciate it a lot. I plan on doing another one about how to set up a VPN and using this VM as a remote gaming platform for our newly created machine. I also give a huge thanks to @spetrillo for figuring out that my IOMMU groups were not good.

 

[UdoB]

Of course I did not test it, but it's really looking good. Thank you!