Boot Windows 10 VM from Passedthrough NVMe Controller

[ikecomp]

The question says it all. I tried searching for this specific question but didn't find anything other than an unraid article but the solution looked very "hacky".
At any rate, I currently have a windows 10 VM that I use as my main workstation and video editing machine. Everything is working great so far except my NVMe write speeds appear to be slower than what I was getting with bare metal windows. I believe this might be a linux driver issue but I wanted to test by completely passing through the controller to windows. I don't do anything else on the NVMe disk/controller except use windows (my linux containers and vm's are on a separate disk) so I don't mind dedicating it to my windows 10 VM.

So the main question is can I passthrough the NVMe controller with NVMe drive directly to my windows 10 VM and boot from it as my OS disk? If so, how would I do that because I don't see pci as an option to choose as a boot disk in the VM config.

Please note, I'm not referring to just passing the disk through using disk-id. I want to pass the entire NVMe controller through. All my devices are in separate iommu groups already so no concern there.

 

[dcsapak]

we would have to add an option for 'bootindex' to a passed through device...
the request is sensible, could you open an enhancement request at https://bugzilla.proxmox.com ?

as a workaround you'd have to look at the resulting passthrough commandline (qm showcmd ID --pretty), and and copy the passthrough part to 'args' (qm set ID -args 'some args') and remove the hostpciX part from the config again (qm set ID -delete hostpciX)

 

[ikecomp]

we would have to add an option for 'bootindex' to a passed through device...
the request is sensible, could you open an enhancement request at https://bugzilla.proxmox.com ?

as a workaround you'd have to look at the resulting passthrough commandline (qm showcmd ID --pretty), and and copy the passthrough part to 'args' (qm set ID -args 'some args') and remove the hostpciX part from the config again (qm set ID -delete hostpciX)

Thanks for the quick reply. I submitted Bug 2684 as an enhancement.

As for the workaround the one question I have is it seems like the command would add the pci device to the arguments line but which of the commands would make it available to use the controller as the boot device? Trying to understand how this would look once I boot the machine. Would there be an extra option in the OVMF boot manager to select from?

Thanks again.

 

[dcsapak]

As for the workaround the one question I have is it seems like the command would add the pci device to the arguments line but which of the commands would make it available to use the controller as the boot device? Trying to understand how this would look once I boot the machine. Would there be an extra option in the OVMF boot manager to select from?

there should be a line like

-device 'vfio-pci,host=0000:21:00.1,id=hostpci0,bus=ich9-pcie-port-1,addr=0x0'

and you would have to add the 'bootindex' parameter (the lowest bootindex gets booted by default), and add it to args, e.g. for my example

qm set VMID -args "-device 'vfio-pci,host=0000:21:00.1,id=hostpci0,bus=ich9-pcie-port-1,addr=0x0,bootindex=50'"

and then remove the hostpci entry
(untested though, since i do not have an nvme to spare atm)

 

[ikecomp]

Ok,got it. Will give it a go and report back. Thanks again.

 

[ikecomp]

Looks like it's a not go. I can passthrough the device as PCIe controller when it's not the bootdisk but when I try to pass it through as the boot disk the VM fails to start and I get the error below



My vm command is as follows

iothread is only valid with virtio disk or virtio-scsi-single controller, ignoring
/usr/bin/kvm \
  -id 100 \
  -name Windows10 \
  -chardev 'socket,id=qmp,path=/var/run/qemu-server/100.qmp,server,nowait' \
  -mon 'chardev=qmp,mode=control' \
  -chardev 'socket,id=qmp-event,path=/var/run/qmeventd.sock,reconnect=5' \
  -mon 'chardev=qmp-event,mode=control' \
  -pidfile /var/run/qemu-server/100.pid \
  -daemonize \
  -smbios 'type=1,uuid=1f3be031-3c78-43cd-bacf-c17674f05f0a' \
  -drive 'if=pflash,unit=0,format=raw,readonly,file=/usr/share/pve-edk2-firmware//OVMF_CODE.fd' \
  -drive 'if=pflash,unit=1,format=qcow2,id=drive-efidisk0,file=/mnt/BULK/images/100/vm-100-disk-0.qcow2' \
  -smp '10,sockets=1,cores=10,maxcpus=10' \
  -nodefaults \
  -boot 'menu=on,strict=on,reboot-timeout=1000,splash=/usr/share/qemu-server/bootsplash.jpg' \
  -vga none \
  -nographic \
  -no-hpet \
  -cpu 'host,+pcid,+kvm_pv_unhalt,+kvm_pv_eoi,hv_spinlocks=0x1fff,hv_vapic,hv_time,hv_reset,hv_vpindex,hv_runtime,hv_relaxed,hv_synic,hv_stimer,hv_ipi,kvm=off' \
  -m 10240 \
  -readconfig /usr/share/qemu-server/pve-q35.cfg \
  -device 'vmgenid,guid=6b57b5c3-ea93-495f-a1e1-61493182b6cb' \
  -device 'usb-tablet,id=tablet,bus=ehci.0,port=1' \
  -device 'vfio-pci,host=0000:0f:00.0,id=hostpci0.0,bus=ich9-pcie-port-1,addr=0x0.0,multifunction=on' \
  -device 'vfio-pci,host=0000:0f:00.1,id=hostpci0.1,bus=ich9-pcie-port-1,addr=0x0.1' \
  -device 'vfio-pci,host=0000:0a:00.0,id=hostpci1,bus=ich9-pcie-port-2,addr=0x0,rombar=0' \
  -device 'vfio-pci,host=0000:0b:00.0,id=hostpci2,bus=ich9-pcie-port-3,addr=0x0,rombar=0' \
  -device 'vfio-pci,host=0000:04:00.0,id=hostpci3,bus=ich9-pcie-port-4,addr=0x0,rombar=0' \
  -device 'usb-host,vendorid=0x046d,productid=0xc043,id=usb0' \
  -device 'usb-host,vendorid=0x2a7a,productid=0x8a4f,id=usb1' \
  -chardev 'socket,path=/var/run/qemu-server/100.qga,server,nowait,id=qga0' \
  -device 'virtio-serial,id=qga0,bus=pci.0,addr=0x8' \
  -device 'virtserialport,chardev=qga0,name=org.qemu.guest_agent.0' \
  -iscsi 'initiator-name=iqn.1993-08.org.debian:01:416de5bc2f6' \
  -device 'virtio-scsi-pci,id=scsihw0,bus=pci.0,addr=0x5' \
  -drive 'file=/dev/disk/by-id/ata-SAMSUNG_HD204UI_S2H7J90B407595,if=none,id=drive-scsi2,format=raw,cache=none,aio=native,detect-zeroes=on' \
  -device 'scsi-hd,bus=scsihw0.0,channel=0,scsi-id=0,lun=2,drive=drive-scsi2,id=scsi2,bootindex=200' \
  -rtc 'driftfix=slew,base=localtime' \
  -machine 'type=pc-q35-3.1+pve0' \
  -global 'kvm-pit.lost_tick_policy=discard' \
  -device 'vfio-pci,host=0000:01:00.0,id=hostpci4,bus=ich9-pcie-port-5,addr=0x0,rombar=0,bootindex=50'

I then went back and looked at the error and tried to set it as just a PCI device. This allowed the VM to start without error but it just booted to the EFI shell and did not detect the nvme drive. I also went into the OMVF boot manager and didn't see the drive there as well. Anything else I could try or is it just a situation where I'll have to wait till the feature is added later?

Thanks again

 

[ikecomp]

Figured it out! Proxmox devs shouldn't really need to do much of anything here based on how I got it to work. The only thing that should be required is to update the OVMF file that proxmox uses.

I scoured the net to see if others were able to get this to work and it was actually simple. I stumbled upon the site below which keeps up-to-date versions of OVMF

https://www.kraxel.org/repos/jenkins/edk2/

From there I downloaded the 64 bit version for my system and extracted the files until I got to the "OVMF_CODE-pure-efi.fd" I then backed up the version that was supplied with proxmox and replaced it with the new one. It then booted me to the EFI shell but once I typed "exit" and it brought me to the boot manager I simply went in, chose my drive and now I can boot from my NVMe just using simple PCIe passthrough already built into proxmox.

Moving forward I will probably restore the backed up OVMF that came with proxmox and just supply a custom location in the future to use the OVMF I want directly to my VM as I don't want any proxmox updates to break my setup. I also want to test for a week or so to confirm this installation is stable.

Thanks again for pointing me in the general direction

 

[safcbee]

Figured it out! Proxmox devs shouldn't really need to do much of anything here based on how I got it to work. The only thing that should be required is to update the OVMF file that proxmox uses.

I scoured the net to see if others were able to get this to work and it was actually simple. I stumbled upon the site below which keeps up-to-date versions of OVMF

https://www.kraxel.org/repos/jenkins/edk2/

From there I downloaded the 64 bit version for my system and extracted the files until I got to the "OVMF_CODE-pure-efi.fd" I then backed up the version that was supplied with proxmox and replaced it with the new one. It then booted me to the EFI shell but once I typed "exit" and it brought me to the boot manager I simply went in, chose my drive and now I can boot from my NVMe just using simple PCIe passthrough already built into proxmox.

Moving forward I will probably restore the backed up OVMF that came with proxmox and just supply a custom location in the future to use the OVMF I want directly to my VM as I don't want any proxmox updates to break my setup. I also want to test for a week or so to confirm this installation is stable.

Thanks again for pointing me in the general direction

How has this been working for you?

I have been tinkering with Unraid for a couple of months now and I have given up, it's waaaaaay too buggy and unstable. Nice features, but as you said above, all seem to be kind of hacked together!

 

[ikecomp]

How has this been working for you?

I have been tinkering with Unraid for a couple of months now and I have given up, it's waaaaaay too buggy and unstable. Nice features, but as you said above, all seem to be kind of hacked together!

I actually ran this for a few months with no issue. I have since stopped passing the Nvme drive through as I wanted to use it to host multiple VM files instead of just the windows 10 VM.

One other thing to note is I saw an update to the e2dk/OVMF firmware in a proxmox update so this might actually work out of the box now but haven't tested this personally.

 

[tristank]

This is still a issue. I'm not able to change the boot order in the uefi because the changes dont stick. I'm passing through a nvme controller but even though I made the necessary changes in the uefi I will still get thrown into the uefi shell. Something overwrites the boot order. I also tried editing the boot order with `efibootmgr -o` and bootice but without success. This is really annoying.

@ikecomp I followed your instructions but I'm still unable to change the boot order. To be clear the nvme device is listed in the uefi and I can boot it but I cant change the boot order.

EDIT: I got it working. @dcsapak instructions suffice. You will have add the args accordingly to your iommu group. For me it was:

$ /root/downloads/iommu_viewer.sh | grep Micron Group: 28 0000:06:00.0 Non-Volatile memory controller [0108]: Micron/Crucial Technology Device [c0a9:540a] (rev 01) Driver: vfio-pci $ qm set 101 -args "-device 'vfio-pci,host=0000:06:00.0,id=hostpci0,bus=ich9-pcie-port-1,addr=0x0,bootindex=50'"

I really would prefer a more elegant solution with a option in the gui.

 

[dcsapak]

@tristank in newer pve installations, the 'boot order' window should allow for hostpci devices to be configured as boot devices...

 

[tristank]

Thanks for answering so quickly. Can you please add a note in that regard to the proxmox admin guide?

 

[dcsapak]

Thanks for answering so quickly. Can you please add a note in that regard to the proxmox admin guide?

it is already there?

10.2.14. Device Boot Order​

in https://pve.proxmox.com/pve-docs/pve-admin-guide.html#chapter_virtual_machines

 

[tristank]

it is already there?

10.2.14. Device Boot Order​

in https://pve.proxmox.com/pve-docs/pve-admin-guide.html#chapter_virtual_machines

My bad. It's already there:

This way, the guest would first attempt to boot from the disk scsi0, if that fails, it would go on to attempt network boot from net0, and in case that fails too, finally attempt to boot from a passed through PCIe device (seen as disk in case of NVMe, otherwise tries to launch into an option ROM).

I must have missed it because I was solely focused on the PCIe Passthrough Chapter.

On a different note: Do you know why Proxmox keeps grabbing the NVMe with the nvme driver even though I defined a softdep?

$ echo "softdep nvme pre: vfio vfio_pci" >> /etc/modprobe.d/nvme.conf $ echo "options vfio-pci ids=c0a9:540a" >> /etc/modprobe.d/vfio.conf

I still have to use driverctl in order to force vfio-pci instead of nvme

$ apt install driverctl $ driverctl set-override 0000:06:00.0 vfio-pci

 

[dcsapak]

On a different note: Do you know why Proxmox keeps grabbing the NVMe with the nvme driver even though I defined a softdep?

probably because nvme is built into the kernel and not a module? so it will already be loaded and bound during the initramfs
i guess this can somehow be overridden using a configfile somewhere in /etc/initramfs and calling update-initramfs after, but idk how off the top of my head

but, my guess is that if you do not mount the drive or in some other way access it, it is not really necessary, since we try to unbind the device from its driver
and bind it to vfio-pci on vm start anyway. so i'd simply try without that (it may be necessary though if the device does not like to be unbound from the driver)

 

[Prox815]

I have followed the steps but encountered the following error:

kvm: -device vfio-pci,host=0000:06:00.0,id=hostpci0,bus=ich9-pcie-port-1,addr=0x0,: PCI: slot 0 function 0 not available for vfio-pci, in use by vfio-pci

Below is my configuration:

agent: 1
args: -cpu 'host, +kvm_pv_unhalt, +kvm_pv_eoi,kvm=off' -device 'vfio-pci,host=0000:06:00.0,id=hostpci1,bus=ich9-pcie-port-1,addr=0x0'
bios: ovmf
boot:
Cores: 4
cpu: host, flags=+pcid hostpci0: 0000:03:00,pcie=1 machine: pc-q35-6.0
memory: 8192
name: msWindows net0: virtio=12:EC:5D:23:AB:01,bridge=vmbr0, firewall=1 numa: 0
onboot : 1
ostype: win10
SCsihw: virtio-Scsi-single smbios1: uuid=a37fc018-5b2e-45b5-9bd6-f4cb7485e841 sockets: 1
Vga: qxl
Vmgenid : 27afaf8e- f253-4181 -addb-e6106e2bca44

I think there is some conflicts with my GPU passthrough, but I don't know how to fix it

 

[dcsapak]

use 'ich9-pcie-port-2' instead for your custom argument....

but why don't you add it as normal 'hostpci1' argument ?

it don't see anything you configure that we would not do too...

 

[Prox815]

Oh, sorry for my misunderstanding, I understand what you mean now -
Because in new pve versions, the boot order can be modified through GUI, and there is no need to use 'arg...' to achieve the function.

I have tried with the following configuration:

agent: 1
bios:
boot: order=hostpci0
Cores: 4
cpu: host, flags=+pcid; +spec-ctrl
hostpci0: 0000:06:00,pcie=1,rombar=0
hostpci1: 0000:03:00,pcie=1
hotplug: network, usb
machine: pc-q35-6.0
memory: 8196
name: msWindows
net0: virtio=56:12:09:12:3D: 10,bridge=vmbr0,firewall=1
numa:0
OStype: win10
Scsihw: virtio-scsi-single
smbios1: uuid=f77ab107-a743-4297-b07c-9c2e78907749 sockets: 1
usb0: host=1-11
Vga: qxl
vmgenid: aOe877ca - 01 c8-4a96-a330 -b2cf8dd7a6ea
args: -cpu 'host,+kvm_pv_unhalt, +kvm_pv_eoi,kvm=off"

Somehow, the q35 Bios cannot detect the nvme drive

# lspci -vvv -s 03:00.0
VGA compatible controller: Advanced Micro Devices,
Kernel driver in use: vfio-pci
Kernel modules: amdgpu

# lspci -vvv -s 06:00.0
root@pve: ~# lspci -s 06:00.0
06:00.0 Non-Volatile memory controller: Sandisk Corp WD Black SN750
...
Kernel driver in use: vfio-pci

nano /etc/modprobe.d/vfio. conf
options vfio-pci ids=1002:731f,1002:ab38,15b7:5006

 

[Prox815]

And I have discovered some errors that are closely related to my problem: