PCIE passthrough

[fidays]

Hi I am running Proxmox VE 6.4-6 on a old desktop with Intel i7 - 4790 processor. I have successfully created few LXC container and ubuntu virtual machines. They are working like charm.

I also have created a windows10 - pro vm and it also worked fine. Now I have a GTX 750 card installed in one of the PCIE slots, which I want to passthrough so that I can remote gaming (RDP or parsec) on it. I followed quite a few tutorials form YouTube none of them worked.
Then I though may be I am using a newer version so I followed this tutorial: https://www.reddit.com/r/Proxmox/comments/lcnn5w/proxmox_pcie_passthrough_in_2_minutes/

This too did not work for me. The issue is when I start my win10 VM, it would not start. The start Icon greys out for few seconds and then again comes back on.
I am not sure what I am doing wrong. I have kept a monitor plugged in to the GPU card but nothing really happens there too.
I am not sure it this issue is with proxmox or with my VM setup.
Either ways, I am stuck with it for about a week now. Any help is much appreciated.


Thank you.

 

[leesteken]

Any information about what went wrong will help. Is there any information in the Task Log or in journalctl -b 0?
Is the GTX 750 the only GPU in the system (or is it used during boot)? video=vesafb:off,efifb:off does not work, you need to use video=vesafb:off video=efifb:off and make sure no nvidia or nouveau or nvidiafb drivers are loaded. Otherwise, you might not need those. When you use this correctly, where won't be any display during startup and it makes it really hard to debug problems. Please don't enable autostart for the VM just yet. Do you have a reliable SSH connection to Proxmox from another computer?
For NVidia cards you should enable Primary GPU and probably set display to None. Sometimes you need to provide a ROM-file (which sometimes needs to be patched), I'm not experienced with NVidia, sorry.
You are breaking up all IOMMU groups and multi-function devices with pcie_acs_overrride, but this is not make passthrough if the GPU is not isolated in a group. Did you really need this? What were your IOMMUI groups like without the overrride? Can you show us using this command: for d in /sys/kernel/iommu_groups/*/devices/*; do n=${d#*/iommu_groups/*}; n=${n%%/*}; printf 'IOMMU group %s ' "$n"; lspci -nns "${d##*/}"; done Please copy-paste it as (inline) code instead of a screenshot for better readability.

 

[fidays]

Thank you for your quick response. It is much appreciated


At 9:15 (GMP+5:30) in the morning I tried to start the Win10 VM again. Here is the output of journalctl -b 0

Jun 03 09:15:17 pmx2 pvedaemon[1107]: <root@pam> starting task UPID:pmx2:00005EF4:00449CB5:60B8504D:qmstart:102:root@pam:
Jun 03 09:15:17 pmx2 pvedaemon[24308]: start VM 102: UPID:pmx2:00005EF4:00449CB5:60B8504D:qmstart:102:root@pam:
Jun 03 09:15:17 pmx2 systemd[1]: Started 102.scope.
Jun 03 09:15:17 pmx2 systemd-udevd[24346]: Using default interface naming scheme 'v240'.
Jun 03 09:15:17 pmx2 systemd-udevd[24346]: link_config: autonegotiation is unset or enabled, the speed and duplex are not writable.
Jun 03 09:15:17 pmx2 systemd-udevd[24346]: Could not generate persistent MAC address for tap102i0: No such file or directory
Jun 03 09:15:18 pmx2 kernel: device tap102i0 entered promiscuous mode
Jun 03 09:15:18 pmx2 systemd-udevd[24346]: link_config: autonegotiation is unset or enabled, the speed and duplex are not writable.
Jun 03 09:15:18 pmx2 systemd-udevd[24346]: Could not generate persistent MAC address for fwbr102i0: No such file or directory
Jun 03 09:15:18 pmx2 systemd-udevd[24345]: link_config: autonegotiation is unset or enabled, the speed and duplex are not writable.
Jun 03 09:15:18 pmx2 systemd-udevd[24345]: Using default interface naming scheme 'v240'.
Jun 03 09:15:18 pmx2 systemd-udevd[24316]: link_config: autonegotiation is unset or enabled, the speed and duplex are not writable.
Jun 03 09:15:18 pmx2 systemd-udevd[24316]: Using default interface naming scheme 'v240'.
Jun 03 09:15:18 pmx2 systemd-udevd[24345]: Could not generate persistent MAC address for fwpr102p0: No such file or directory
Jun 03 09:15:18 pmx2 systemd-udevd[24316]: Could not generate persistent MAC address for fwln102i0: No such file or directory
Jun 03 09:15:18 pmx2 kernel: fwbr102i0: port 1(fwln102i0) entered blocking state
Jun 03 09:15:18 pmx2 kernel: fwbr102i0: port 1(fwln102i0) entered disabled state
Jun 03 09:15:18 pmx2 kernel: device fwln102i0 entered promiscuous mode
Jun 03 09:15:18 pmx2 kernel: fwbr102i0: port 1(fwln102i0) entered blocking state
Jun 03 09:15:18 pmx2 kernel: fwbr102i0: port 1(fwln102i0) entered forwarding state
Jun 03 09:15:18 pmx2 kernel: vmbr0: port 3(fwpr102p0) entered blocking state
Jun 03 09:15:18 pmx2 kernel: vmbr0: port 3(fwpr102p0) entered disabled state
Jun 03 09:15:18 pmx2 kernel: device fwpr102p0 entered promiscuous mode
Jun 03 09:15:18 pmx2 kernel: vmbr0: port 3(fwpr102p0) entered blocking state
Jun 03 09:15:18 pmx2 kernel: vmbr0: port 3(fwpr102p0) entered forwarding state
Jun 03 09:15:18 pmx2 kernel: fwbr102i0: port 2(tap102i0) entered blocking state
Jun 03 09:15:18 pmx2 kernel: fwbr102i0: port 2(tap102i0) entered disabled state
Jun 03 09:15:18 pmx2 kernel: fwbr102i0: port 2(tap102i0) entered blocking state
Jun 03 09:15:18 pmx2 kernel: fwbr102i0: port 2(tap102i0) entered forwarding state
Jun 03 09:15:18 pmx2 kernel: fwbr102i0: port 2(tap102i0) entered disabled state
Jun 03 09:15:18 pmx2 kernel: fwbr102i0: port 1(fwln102i0) entered disabled state
Jun 03 09:15:18 pmx2 kernel: vmbr0: port 3(fwpr102p0) entered disabled state
Jun 03 09:15:18 pmx2 kernel: device fwln102i0 left promiscuous mode
Jun 03 09:15:18 pmx2 kernel: fwbr102i0: port 1(fwln102i0) entered disabled state
Jun 03 09:15:18 pmx2 kernel: device fwpr102p0 left promiscuous mode
Jun 03 09:15:18 pmx2 kernel: vmbr0: port 3(fwpr102p0) entered disabled state
Jun 03 09:15:18 pmx2 pvedaemon[1106]: VM 102 qmp command failed - VM 102 not running
Jun 03 09:15:18 pmx2 systemd[1]: 102.scope: Succeeded.
Jun 03 09:15:18 pmx2 pvedaemon[24308]: start failed: QEMU exited with code 1
Jun 03 09:15:18 pmx2 pvedaemon[1107]: <root@pam> end task UPID:pmx2:00005EF4:00449CB5:60B8504D:qmstart:102:root@pam: start failed: QEMU exited with code 1
lines 4393-4441/4441 (END)

• Yes GTX 750 is one only GPU present in the system
• I have disabled the NVIDIA driver from host, so there are no display for the host.
• I use the web interface and ssh to manage the host
• VM is not setup to start at boot

I did not really gave a though about pcie_acs_overrride as I was blindly following a tutorial to get this done.
Here is the output of the code you shared.

root@pmx2:~# for d in /sys/kernel/iommu_groups/*/devices/*; do n=${d#*/iommu_groups/*}; n=${n%%/*}; printf 'IOMMU group %s ' "$n"; lspci -nns "${d##*/}"; done
IOMMU group 0 00:00.0 Host bridge [0600]: Intel Corporation 4th Gen Core Processor DRAM Controller [8086:0c00] (rev 06)
IOMMU group 10 00:1c.3 PCI bridge [0604]: Intel Corporation 9 Series Chipset Family PCI Express Root Port 4 [8086:8c96] (rev d0)
IOMMU group 11 00:1c.4 PCI bridge [0604]: Intel Corporation 9 Series Chipset Family PCI Express Root Port 5 [8086:8c98] (rev d0)
IOMMU group 12 00:1d.0 USB controller [0c03]: Intel Corporation 9 Series Chipset Family USB EHCI Controller #1 [8086:8ca6]
IOMMU group 13 00:1f.0 ISA bridge [0601]: Intel Corporation 9 Series Chipset Family H97 Controller [8086:8cc6]
IOMMU group 13 00:1f.2 SATA controller [0106]: Intel Corporation 9 Series Chipset Family SATA Controller [AHCI Mode] [8086:8c82]
IOMMU group 13 00:1f.3 SMBus [0c05]: Intel Corporation 9 Series Chipset Family SMBus Controller [8086:8ca2]
IOMMU group 14 01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GM107 [GeForce GTX 750] [10de:1381] (rev a2)       
IOMMU group 15 01:00.1 Audio device [0403]: NVIDIA Corporation Device [10de:0fbc] (rev a1)
IOMMU group 16 03:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller [10ec:8168] (rev 0c)
IOMMU group 17 04:00.0 PCI bridge [0604]: Intel Corporation 82801 PCI Bridge [8086:244e] (rev 41)
IOMMU group 18 06:00.0 Network controller [0280]: Realtek Semiconductor Co., Ltd. RTL8812AE 802.11ac PCIe Wireless Network Adapter [10ec:8812] (rev 01)
IOMMU group 1 00:01.0 PCI bridge [0604]: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x16 Controller [8086:0c01] (rev 06)     
IOMMU group 2 00:02.0 VGA compatible controller [0300]: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor Integrated Graphics Controller [8086:0412] (rev 06)
IOMMU group 3 00:03.0 Audio device [0403]: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor HD Audio Controller [8086:0c0c] (rev 06)
IOMMU group 4 00:14.0 USB controller [0c03]: Intel Corporation 9 Series Chipset Family USB xHCI Controller [8086:8cb1]
IOMMU group 5 00:16.0 Communication controller [0780]: Intel Corporation 9 Series Chipset Family ME Interface #1 [8086:8cba]
IOMMU group 6 00:1a.0 USB controller [0c03]: Intel Corporation 9 Series Chipset Family USB EHCI Controller #2 [8086:8cad]   
IOMMU group 7 00:1b.0 Audio device [0403]: Intel Corporation 9 Series Chipset Family HD Audio Controller [8086:8ca0]
IOMMU group 8 00:1c.0 PCI bridge [0604]: Intel Corporation 9 Series Chipset Family PCI Express Root Port 1 [8086:8c90] (rev d0)
IOMMU group 9 00:1c.2 PCI bridge [0604]: Intel Corporation 9 Series Chipset Family PCI Express Root Port 3 [8086:8c94] (rev d0)

Also cloud you help me with the VM configuration. I was confused about how to configure graphic card section while creating the VM. As of now I have left it default.



Also, is it true that the VM has to be created after the host configuration is done. I read it somewhere that once the VM is created, the passthrough is not possible due to some CPU setting gets created only once during VM creation and can not be changed later.

I will try your suggestion on the GRUB file.

 

[leesteken]

Your Xeon has an integrated GPU also. Can you use that one during boot for the Proxmox host? That keeps the BIOS from touching the GTX 750, which makes passthrough easier. Then you also don't need any of the kernel paramters after iommu=pt (which is also not strictly needed). Is you system booting with GRUB or systemd?
Any setting of the VM can be changed later. Can you show us the VM configuration from the file /etc/pve/qemu-server/102.conf (instead of the GUI)? There are changes that require a new install of Windows (like switching between SeaBIOS and OVMF, I think). For VirtIO devices, you need Windows drivers.
Unfortunately no error information in the system log. Was there any information in the task log (from double clicking Tasks in the bottom of the web GUI)? Did the Windows VM work fine before the passthrough?

 

[fidays]

1. I think GRUB
2.

root@pmx2:~# cat /etc/pve/qemu-server/102.conf
bios: ovmf
boot: order=scsi0;ide2;net0
cores: 2
efidisk0: ssd-pool-pmx2:vm-102-disk-1,size=4M
hostpci0: 01:00
hostpci1: 01:00
ide2: nas-global:iso/Windows10_x64.iso,media=cdrom
machine: pc-q35-5.2
memory: 8196
name: win10
net0: e1000=6A:D3:A8:FF:6D:85,bridge=vmbr0,firewall=1
numa: 0
ostype: win10
scsi0: ssd-pool-pmx2:vm-102-disk-0,cache=writethrough,size=60G,ssd=1
scsihw: virtio-scsi-pci
smbios1: uuid=190ea724-78a8-46c9-9a12-9d2967c92af0
sockets: 1
vmgenid: 35bf6c9e-cfc4-4c9b-b31c-cc28e130301f

3. Task output:

kvm: -device vfio-pci,host=0000:01:00.0,id=hostpci1.0,bus=pci.0,addr=0x11.0,multifunction=on: vfio 0000:01:00.0: device is already attached TASK ERROR: start failed: QEMU exited with code 1


4. Here is my blacklist file

root@pmx2:~# cat /etc/modprobe.d/blacklist.conf
blacklist radeon
blacklist nouveau
blacklist nvidia

 

[leesteken]

You have the same device specified twice in the configuration file:
hostpci0: 01:00 hostpci1: 01:00
I think that explains the error message device is already attached. Please remove one of those lines.

 

[fidays]

I have removed one PCIE device (nvidia audio) and it now boots into the logo. but still has an error. I think the EUFI disk is not bootable.
Please can you tell whats might be causing this.

 

[leesteken]

Sorry for being pedantic, but you did not "remove the audio device", you pass both devices because you selected multi-fumction, which is correct. But you must not add it twice to the VM.
Did you switch from SeaBios to OVMF or change the machine type? Windows cannot handle that and needs to be reinstalled. Otherwise, make sure to select the correct bootable disk. Are you seeing this on the display connected to the GTX 750 or are you still using a virtual display as well. Please also select Primary GPU (or add ,x-vga=on) and select Display: none (or set vga: none).

 

[fidays]

No it was in the vnc. Now I have set the display to none, changed the boot order. I am still getting the same screen, only difference is, that now this is on GPU monitor.

I am using KVM64 as virtual processor.
My host processor is i7 - 4790. I have gone ahead in host bios and enabled VT-D
but in my VM config if I set the processor to be Broadwell, It fails to start.This is the msg.



What should I set my CPU to and what to do suggest to solve the uefi boot issue?

 

[leesteken]

Good, PCI passthrough is working. Now just fix your WIndows installation or reinstall. I don't know how to fix Windows boot issues, maybe someone else knows?
You have a Haswell generation processor, ofcourse it does not support all features of Broadwell generation. Why not set it host?

 

[fidays]

Its a UEFI issue, when i changed the cd media to point at a ubuntu.iso it worked.
I do not know how to make the windows10_x64.iso boot with UEFI

 

[fidays]

Good, PCI passthrough is working. Now just fix your WIndows installation or reinstall. I don't know how to fix Windows boot issues, maybe someone else knows?
You have a Haswell generation processor, ofcourse it does not support all features of Broadwell generation. Why not set it host?

Ok i will set it to host

 

[fidays]

Got the issue. During boot it is asking "Press any key to boot from CD" as there are no keyboards passed through, I cant do that so it is timing out and going into UEFI shell.

Can you recommend any Tutorial link for USB passthrough?

 

[leesteken]

Its a UEFI issue, when i changed the cd media to point at a ubuntu.iso it worked.
I do not know how to make the windows10_x64.iso boot with UEFI

I also don't know. I just select IDE2 as boot devices in VM settings (or I press Escape during the boot screen with Promox logo) and press a key when it says "Press any key to boot from DVD". Because you are no longer using VNC, you might need to pass a USB controller (or port) to have a keyboard working in the VM. Maybe it is easier to remove the GTX 750 from the VM and go back to Display VGA to reinstall Windows with VNC, and when that works add the GTX 750 to the VM?

 

[leesteken]

For best performance passthrough your 00:14.0 USB controller (as well as the GTX 750) and plugin the keyboard and mouse.

 

[fidays]

Solved.
Thank you AVW. It has been a great help. This is really appreciated.