PCIe Passthrough

[dabaers]

I have seen the many threads but I promise I'm a special case here. I have Proxmox VE installed with ubuntu and windows VM which I have two different tasks. Long story short I can physically see my graphics card as a passthrough option but once I go to initialize the VM doesn't work. I have attempted to have a monitor plugged into the card but here is where we get to the rabbit hole. I cant check Iommu status with dmesg | grep -e DMAR -e IOMMU additionally grub.d as a directory is blank. etc/default/grub
GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on pcie_acs_override=downstream,multifunction video=efifbff"

The server is running an e5 2650 v3, Asrock x99 extreme 4 and a gtx1050 for passthrough.

Im relatively new to all of this and any help would be awesome!

 

[Stefan_R]

I cant check Iommu status with dmesg | grep -e DMAR -e IOMMU

Uhm... why not? Type it in, run it, and post what it says?

doesn't work

Tip for getting help online in general: whenever your tempted to write "doesn't work" without any context, rewrite your sentence with context

What exactly doesn't work? What are your expecting to happen, and what happens? What do the error messages say, if there are any, what shows up in syslog (journalctl -e)?

You're "GRUB_CMDLINE_LINUX_DEFAULT" looks correct, though it's a good idea to add 'iommu=pt' as well for improved host performance. Did you bind the vfio-pci driver to your GPU as indicated on our wiki?

 

[dabaers]

This is what happens when I try to check iommu.

root@Homestead:~# dmesg | grep -e DMAR -e IOMMU
root@Homestead:~#

When I try to initialize the VM the will attempt to start then the log comes back with a error - and hard freezes the VM to the point I have to restart the entire node.

Here is the vfio in /etc/modules
# /etc/modules: kernel modules to load at boot time.
#
# This file contains the names of kernel modules that should be loaded
# at boot time, one per line. Lines beginning with "#" are ignored.

vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd


here is the system log

Oct 12 12:28:28 Homestead kernel: vfio-pci 0000:02:00.0: AER: TLP Header: 400>
Oct 12 12:28:28 Homestead kernel: pcieport 0000:00:03.0: AER: device recovery s>
Oct 12 12:28:28 Homestead kernel: pcieport 0000:00:03.0: AER: Multiple Uncorrec>
Oct 12 12:28:28 Homestead kernel: vfio-pci 0000:02:00.0: PCIe Bus Error: severi>
Oct 12 12:28:28 Homestead kernel: vfio-pci 0000:02:00.0: device [10de:1c81] e>
Oct 12 12:28:28 Homestead kernel: vfio-pci 0000:02:00.0: [20] UnsupReq >
Oct 12 12:28:28 Homestead kernel: vfio-pci 0000:02:00.0: AER: TLP Header: 400>
Oct 12 12:28:28 Homestead kernel: pcieport 0000:00:03.0: AER: device recovery s>
Oct 12 12:28:28 Homestead kernel: pcieport 0000:00:03.0: AER: Multiple Uncorrec>
Oct 12 12:28:28 Homestead kernel: vfio-pci 0000:02:00.0: PCIe Bus Error: severi>
Oct 12 12:28:28 Homestead kernel: vfio-pci 0000:02:00.0: device [10de:1c81] e>
Oct 12 12:28:28 Homestead kernel: vfio-pci 0000:02:00.0: [20] UnsupReq >
Oct 12 12:28:28 Homestead kernel: vfio-pci 0000:02:00.0: AER: TLP Header: 400>
Oct 12 12:28:28 Homestead kernel: pcieport 0000:00:03.0: AER: device recovery s>
Oct 12 12:28:28 Homestead kernel: pcieport 0000:00:03.0: AER: Multiple Uncorrec>
Oct 12 12:28:28 Homestead kernel: vfio-pci 0000:02:00.0: PCIe Bus Error: severi>
Oct 12 12:28:28 Homestead kernel: vfio-pci 0000:02:00.0: device [10de:1c81] e>
Oct 12 12:28:28 Homestead kernel: vfio-pci 0000:02:00.0: [20] UnsupReq >
Oct 12 12:28:28 Homestead kernel: vfio-pci 0000:02:00.0: AER: TLP Header: 400>
Oct 12 12:28:28 Homestead kernel: pcieport 0000:00:03.0: AER: device recovery s>
Oct 12 12:28:28 Homestead kernel: pcieport 0000:00:03.0: AER: Multiple Uncorrec>
Oct 12 12:28:28 Homestead kernel: vfio-pci 0000:02:00.0: PCIe Bus Error: severi>
Oct 12 12:28:28 Homestead kernel: vfio-pci 0000:02:00.0: device [10de:1c81] e>
lines 979-1001/1001 (END)
Oct 12 12:28:28 Homestead kernel: vfio-pci 0000:02:00.0: AER: TLP Header: 40000008 000002ff f1030000 f7f7f7f7
Oct 12 12:28:28 Homestead kernel: pcieport 0000:00:03.0: AER: device recovery successful
Oct 12 12:28:28 Homestead kernel: pcieport 0000:00:03.0: AER: Multiple Uncorrected (Non-Fatal) error received: 0000:02:00.0
Oct 12 12:28:28 Homestead kernel: vfio-pci 0000:02:00.0: PCIe Bus Error: severity=Uncorrected (Non-Fatal), type=Transaction Layer, (Requester ID)
Oct 12 12:28:28 Homestead kernel: vfio-pci 0000:02:00.0: device [10de:1c81] error status/mask=00100000/00000000
Oct 12 12:28:28 Homestead kernel: vfio-pci 0000:02:00.0: [20] UnsupReq (First)
Oct 12 12:28:28 Homestead kernel: vfio-pci 0000:02:00.0: AER: TLP Header: 40000008 000002ff f1030000 f7f7f7f7
Oct 12 12:28:28 Homestead kernel: pcieport 0000:00:03.0: AER: device recovery successful
Oct 12 12:28:28 Homestead kernel: pcieport 0000:00:03.0: AER: Multiple Uncorrected (Non-Fatal) error received: 0000:02:00.0
Oct 12 12:28:28 Homestead kernel: vfio-pci 0000:02:00.0: PCIe Bus Error: severity=Uncorrected (Non-Fatal), type=Transaction Layer, (Requester ID)
Oct 12 12:28:28 Homestead kernel: vfio-pci 0000:02:00.0: device [10de:1c81] error status/mask=00100000/00000000
Oct 12 12:28:28 Homestead kernel: vfio-pci 0000:02:00.0: [20] UnsupReq (First)
Oct 12 12:28:28 Homestead kernel: vfio-pci 0000:02:00.0: AER: TLP Header: 40000008 000002ff f1030000 f7f7f7f7
Oct 12 12:28:28 Homestead kernel: pcieport 0000:00:03.0: AER: device recovery successful
Oct 12 12:28:28 Homestead kernel: pcieport 0000:00:03.0: AER: Multiple Uncorrected (Non-Fatal) error received: 0000:02:00.0
Oct 12 12:28:28 Homestead kernel: vfio-pci 0000:02:00.0: PCIe Bus Error: severity=Uncorrected (Non-Fatal), type=Transaction Layer, (Requester ID)
Oct 12 12:28:28 Homestead kernel: vfio-pci 0000:02:00.0: device [10de:1c81] error status/mask=00100000/00000000
Oct 12 12:28:28 Homestead kernel: vfio-pci 0000:02:00.0: [20] UnsupReq (First)
Oct 12 12:28:28 Homestead kernel: vfio-pci 0000:02:00.0: AER: TLP Header: 40000008 000002ff f1030000 f7f7f7f7
Oct 12 12:28:28 Homestead kernel: pcieport 0000:00:03.0: AER: device recovery successful
Oct 12 12:28:28 Homestead kernel: pcieport 0000:00:03.0: AER: Multiple Uncorrected (Non-Fatal) error received: 0000:02:00.0
Oct 12 12:28:28 Homestead kernel: vfio-pci 0000:02:00.0: PCIe Bus Error: severity=Uncorrected (Non-Fatal), type=Transaction Layer, (Requester ID)
Oct 12 12:28:28 Homestead kernel: vfio-pci 0000:02:00.0: device [10de:1c81] error status/mask=00100000/00000000

I appreciate the help.

 

[leesteken]

I see a lot of AER: Multiple Uncorrected (Non-Fatal) error received. Is this the only GPU in your system? Does it display the startup of the Proxmox host? It often help to add the following kernel parameters: nomodeset textonly video=vesafb:off video=efifb:off.

 

[dabaers]

I see a lot of AER: Multiple Uncorrected (Non-Fatal) error received. Is this the only GPU in your system? Does it display the startup of the Proxmox host? It often help to add the following kernel parameters: nomodeset textonly video=vesafb:off video=efifb:off.

Yes it is the only gpu in the system, I get a Qemu exited with code 1. also I added that line of code to grub and still the same problem. Sidenote if its important the Vm is windows based and I followed this guide: https://www.youtube.com/watch?v=fgx3NMk6F54&t=453s - Clearly not well enough.

 

[dabaers]

So Update: I switched to bios 2 as the motherboard supports it, the VM boots but the qemu/guest agent doesn't work for RDC. I'm not sure if it's hanging somewhere in the boot or if this is a side effect of something else altogether. Additionally, I noticed that an option of ACS from bios level needs to be enabled for Xeon e3 and e5 processors, is this something that could affect it? I haven't been able to find the option in the bios to enable or disable.

 

[leesteken]

What is the difference between bios 2 and bios 1?
In principle, you want ACS enabled because it determines the IOMMU groups. But you use pcie_acs_override to ignore ACS and split those groups up. Do you really need that ACS override? Did you check if the GPU was in an IOMMU group without other devices (except bridges) before using the override?

 

[dabaers]

What is the difference between bios 2 and bios 1?
In principle, you want ACS enabled because it determines the IOMMU groups. But you use pcie_acs_override to ignore ACS and split those groups up. Do you really need that ACS override? Did you check if the GPU was in an IOMMU group without other devices (except bridges) before using the override?

Same version of bios just wanted to see if there was a sticking point somewhere. Also how would I check iommu groups? I have pulled it up before and seen the lines of code but how do you determine groups?

 

[leesteken]

Same version of bios just wanted to see if there was a sticking point somewhere. Also how would I check iommu groups? I have pulled it up before and seen the lines of code but how do you determine groups?

dmesg | grep iommu, for example, shows all devices added to each IOMMU group. Each group has a unique number. Devices are in the same group when they are added to the group with the same number. For a fancy overview use: for d in /sys/kernel/iommu_groups/*/devices/*; do n=${d#*/iommu_groups/*}; n=${n%%/*}; printf 'IOMMU group %s ' "$n"; lspci -nnks "${d##*/}"; done. Make sure to look at the whole group that contains the device you want to passthrough.

 

[dabaers]

hey so here is the group list. Im not sure if I don't know what I am doing but the VGA 02.00.0 in group 30 I think is the graphics card because that is the correct number for it

I had to use find sys/kernel/iommu_groups as dmesg | grep iommu did not produce a result

/sys/kernel/iommu_groups/17
/sys/kernel/iommu_groups/17/devices
/sys/kernel/iommu_groups/17/devices/0000:00:05.2
/sys/kernel/iommu_groups/17/type
/sys/kernel/iommu_groups/17/reserved_regions
/sys/kernel/iommu_groups/7
/sys/kernel/iommu_groups/7/devices
/sys/kernel/iommu_groups/7/devices/0000:ff:14.7
/sys/kernel/iommu_groups/7/devices/0000:ff:14.5
/sys/kernel/iommu_groups/7/devices/0000:ff:14.3
/sys/kernel/iommu_groups/7/devices/0000:ff:14.1
/sys/kernel/iommu_groups/7/devices/0000:ff:14.6
/sys/kernel/iommu_groups/7/devices/0000:ff:14.4
/sys/kernel/iommu_groups/7/devices/0000:ff:14.2
/sys/kernel/iommu_groups/7/devices/0000:ff:14.0
/sys/kernel/iommu_groups/7/type
/sys/kernel/iommu_groups/7/reserved_regions
/sys/kernel/iommu_groups/25
/sys/kernel/iommu_groups/25/devices
/sys/kernel/iommu_groups/25/devices/0000:00:1b.0
/sys/kernel/iommu_groups/25/type
/sys/kernel/iommu_groups/25/reserved_regions
/sys/kernel/iommu_groups/15
/sys/kernel/iommu_groups/15/devices
/sys/kernel/iommu_groups/15/devices/0000:00:05.0
/sys/kernel/iommu_groups/15/type
/sys/kernel/iommu_groups/15/reserved_regions
/sys/kernel/iommu_groups/5
/sys/kernel/iommu_groups/5/devices
/sys/kernel/iommu_groups/5/devices/0000:ff:12.4
/sys/kernel/iommu_groups/5/devices/0000:ff:12.0
/sys/kernel/iommu_groups/5/devices/0000:ff:12.5
/sys/kernel/iommu_groups/5/devices/0000:ff:12.1
/sys/kernel/iommu_groups/5/type
/sys/kernel/iommu_groups/5/reserved_regions
/sys/kernel/iommu_groups/23
/sys/kernel/iommu_groups/23/devices
/sys/kernel/iommu_groups/23/devices/0000:00:19.0
/sys/kernel/iommu_groups/23/type
/sys/kernel/iommu_groups/23/reserved_regions
/sys/kernel/iommu_groups/13
/sys/kernel/iommu_groups/13/devices
/sys/kernel/iommu_groups/13/devices/0000:00:01.0
/sys/kernel/iommu_groups/13/type
/sys/kernel/iommu_groups/13/reserved_regions
/sys/kernel/iommu_groups/31
/sys/kernel/iommu_groups/31/devices
/sys/kernel/iommu_groups/31/devices/0000:02:00.1
/sys/kernel/iommu_groups/31/type
/sys/kernel/iommu_groups/31/reserved_regions
/sys/kernel/iommu_groups/3
/sys/kernel/iommu_groups/3/devices
/sys/kernel/iommu_groups/3/devices/0000:ff:0f.6
/sys/kernel/iommu_groups/3/devices/0000:ff:0f.4
/sys/kernel/iommu_groups/3/devices/0000:ff:0f.2
/sys/kernel/iommu_groups/3/devices/0000:ff:0f.0
/sys/kernel/iommu_groups/3/devices/0000:ff:0f.5
/sys/kernel/iommu_groups/3/devices/0000:ff:0f.3
/sys/kernel/iommu_groups/3/devices/0000:ff:0f.1
/sys/kernel/iommu_groups/3/type
/sys/kernel/iommu_groups/3/reserved_regions
/sys/kernel/iommu_groups/21
/sys/kernel/iommu_groups/21/devices
/sys/kernel/iommu_groups/21/devices/0000:00:14.0
/sys/kernel/iommu_groups/21/type
/sys/kernel/iommu_groups/21/reserved_regions
/sys/kernel/iommu_groups/11
/sys/kernel/iommu_groups/11/devices
/sys/kernel/iommu_groups/11/devices/0000:ff:1f.0
/sys/kernel/iommu_groups/11/devices/0000:ff:1f.2
/sys/kernel/iommu_groups/11/type
/sys/kernel/iommu_groups/11/reserved_regions
/sys/kernel/iommu_groups/1
/sys/kernel/iommu_groups/1/devices
/sys/kernel/iommu_groups/1/devices/0000:ff:0c.6
/sys/kernel/iommu_groups/1/devices/0000:ff:0c.4
/sys/kernel/iommu_groups/1/devices/0000:ff:0c.2
/sys/kernel/iommu_groups/1/devices/0000:ff:0c.0
/sys/kernel/iommu_groups/1/devices/0000:ff:0c.7
/sys/kernel/iommu_groups/1/devices/0000:ff:0c.5
/sys/kernel/iommu_groups/1/devices/0000:ff:0c.3
/sys/kernel/iommu_groups/1/devices/0000:ff:0c.1
/sys/kernel/iommu_groups/1/type
/sys/kernel/iommu_groups/1/reserved_regions
/sys/kernel/iommu_groups/28
/sys/kernel/iommu_groups/28/devices
/sys/kernel/iommu_groups/28/devices/0000:00:1d.0
/sys/kernel/iommu_groups/28/type
/sys/kernel/iommu_groups/28/reserved_regions
/sys/kernel/iommu_groups/18
/sys/kernel/iommu_groups/18/devices
/sys/kernel/iommu_groups/18/devices/0000:00:05.4
/sys/kernel/iommu_groups/18/type
/sys/kernel/iommu_groups/18/reserved_regions
/sys/kernel/iommu_groups/8
/sys/kernel/iommu_groups/8/devices
/sys/kernel/iommu_groups/8/devices/0000:ff:16.2
/sys/kernel/iommu_groups/8/devices/0000:ff:16.0
/sys/kernel/iommu_groups/8/devices/0000:ff:16.7
/sys/kernel/iommu_groups/8/devices/0000:ff:16.3
/sys/kernel/iommu_groups/8/devices/0000:ff:16.1
/sys/kernel/iommu_groups/8/devices/0000:ff:16.6
/sys/kernel/iommu_groups/8/type
/sys/kernel/iommu_groups/8/reserved_regions
/sys/kernel/iommu_groups/26
/sys/kernel/iommu_groups/26/devices
/sys/kernel/iommu_groups/26/devices/0000:00:1c.0
/sys/kernel/iommu_groups/26/type
/sys/kernel/iommu_groups/26/reserved_regions
/sys/kernel/iommu_groups/16
/sys/kernel/iommu_groups/16/devices
/sys/kernel/iommu_groups/16/devices/0000:00:05.1
/sys/kernel/iommu_groups/16/type
/sys/kernel/iommu_groups/16/reserved_regions
/sys/kernel/iommu_groups/6
/sys/kernel/iommu_groups/6/devices
/sys/kernel/iommu_groups/6/devices/0000:ff:13.2
/sys/kernel/iommu_groups/6/devices/0000:ff:13.0
/sys/kernel/iommu_groups/6/devices/0000:ff:13.7
/sys/kernel/iommu_groups/6/devices/0000:ff:13.3
/sys/kernel/iommu_groups/6/devices/0000:ff:13.1
/sys/kernel/iommu_groups/6/devices/0000:ff:13.6
/sys/kernel/iommu_groups/6/type
/sys/kernel/iommu_groups/6/reserved_regions
/sys/kernel/iommu_groups/24
/sys/kernel/iommu_groups/24/devices
/sys/kernel/iommu_groups/24/devices/0000:00:1a.0
/sys/kernel/iommu_groups/24/type
/sys/kernel/iommu_groups/24/reserved_regions
/sys/kernel/iommu_groups/14
/sys/kernel/iommu_groups/14/devices
/sys/kernel/iommu_groups/14/devices/0000:00:03.0
/sys/kernel/iommu_groups/14/type
/sys/kernel/iommu_groups/14/reserved_regions
/sys/kernel/iommu_groups/32
/sys/kernel/iommu_groups/32/devices
/sys/kernel/iommu_groups/32/devices/0000:04:00.0
/sys/kernel/iommu_groups/32/type
/sys/kernel/iommu_groups/32/reserved_regions
/sys/kernel/iommu_groups/4
/sys/kernel/iommu_groups/4/devices
/sys/kernel/iommu_groups/4/devices/0000:ff:10.0
/sys/kernel/iommu_groups/4/devices/0000:ff:10.7
/sys/kernel/iommu_groups/4/devices/0000:ff:10.5
/sys/kernel/iommu_groups/4/devices/0000:ff:10.1
/sys/kernel/iommu_groups/4/devices/0000:ff:10.6
/sys/kernel/iommu_groups/4/type
/sys/kernel/iommu_groups/4/reserved_regions
/sys/kernel/iommu_groups/22
/sys/kernel/iommu_groups/22/devices
/sys/kernel/iommu_groups/22/devices/0000:00:16.0
/sys/kernel/iommu_groups/22/type
/sys/kernel/iommu_groups/22/reserved_regions
/sys/kernel/iommu_groups/12
/sys/kernel/iommu_groups/12/devices
/sys/kernel/iommu_groups/12/devices/0000:00:00.0
/sys/kernel/iommu_groups/12/type
/sys/kernel/iommu_groups/12/reserved_regions
/sys/kernel/iommu_groups/30
/sys/kernel/iommu_groups/30/devices
/sys/kernel/iommu_groups/30/devices/0000:02:00.0
/sys/kernel/iommu_groups/30/type
/sys/kernel/iommu_groups/30/reserved_regions
/sys/kernel/iommu_groups/2
/sys/kernel/iommu_groups/2/devices
/sys/kernel/iommu_groups/2/devices/0000:ff:0d.1
/sys/kernel/iommu_groups/2/devices/0000:ff:0d.0
/sys/kernel/iommu_groups/2/type
/sys/kernel/iommu_groups/2/reserved_regions
/sys/kernel/iommu_groups/20
/sys/kernel/iommu_groups/20/devices
/sys/kernel/iommu_groups/20/devices/0000:00:11.4
/sys/kernel/iommu_groups/20/type
/sys/kernel/iommu_groups/20/reserved_regions
/sys/kernel/iommu_groups/10
/sys/kernel/iommu_groups/10/devices
/sys/kernel/iommu_groups/10/devices/0000:ff:1e.4
/sys/kernel/iommu_groups/10/devices/0000:ff:1e.2
/sys/kernel/iommu_groups/10/devices/0000:ff:1e.0
/sys/kernel/iommu_groups/10/devices/0000:ff:1e.3
/sys/kernel/iommu_groups/10/devices/0000:ff:1e.1
/sys/kernel/iommu_groups/10/type
/sys/kernel/iommu_groups/10/reserved_regions
/sys/kernel/iommu_groups/29
/sys/kernel/iommu_groups/29/devices
/sys/kernel/iommu_groups/29/devices/0000:00:1f.2
/sys/kernel/iommu_groups/29/devices/0000:00:1f.0
/sys/kernel/iommu_groups/29/devices/0000:00:1f.3
/sys/kernel/iommu_groups/29/type
/sys/kernel/iommu_groups/29/reserved_regions
/sys/kernel/iommu_groups/0
/sys/kernel/iommu_groups/0/devices
/sys/kernel/iommu_groups/0/devices/0000:ff:0b.1
/sys/kernel/iommu_groups/0/devices/0000:ff:0b.2
/sys/kernel/iommu_groups/0/devices/0000:ff:0b.0
/sys/kernel/iommu_groups/0/type
/sys/kernel/iommu_groups/0/reserved_regions
/sys/kernel/iommu_groups/19
/sys/kernel/iommu_groups/19/devices
/sys/kernel/iommu_groups/19/devices/0000:00:11.0
/sys/kernel/iommu_groups/19/type
/sys/kernel/iommu_groups/19/reserved_regions
/sys/kernel/iommu_groups/9
/sys/kernel/iommu_groups/9/devices
/sys/kernel/iommu_groups/9/devices/0000:ff:17.7
/sys/kernel/iommu_groups/9/devices/0000:ff:17.5
/sys/kernel/iommu_groups/9/devices/0000:ff:17.3
/sys/kernel/iommu_groups/9/devices/0000:ff:17.1
/sys/kernel/iommu_groups/9/devices/0000:ff:17.6
/sys/kernel/iommu_groups/9/devices/0000:ff:17.4
/sys/kernel/iommu_groups/9/devices/0000:ff:17.2
/sys/kernel/iommu_groups/9/devices/0000:ff:17.0
/sys/kernel/iommu_groups/9/type
/sys/kernel/iommu_groups/9/reserved_regions
/sys/kernel/iommu_groups/27
/sys/kernel/iommu_groups/27/devices
/sys/kernel/iommu_groups/27/devices/0000:00:1c.3
/sys/kernel/iommu_groups/27/type
/sys/kernel/iommu_groups/27/reserved_regions

 

[dabaers]

Also a fun new addition to this problem is that the kern.log is 17 gb and my local fold is now full. Which I do not know which files I can and should clear and how to do it via SSH because I can no longer Gui my way in

 

[leesteken]

Deleting log files once in a while should be fine: rm /var/log/kern.log. Or maybe removing the old ones is enough? rm /var/log/kern.log.* Or all non-currents logs? rm /var/log/*.log.*

 

[dabaers]

So I've been thinking and wild question here, my server won't boot without a GPU installed, would this create an issue when PCI passthrough is enabled with a permissions or required hardware level problem?

 

[leesteken]

So I've been thinking and wild question here, my server won't boot without a GPU installed, would this create an issue when PCI passthrough is enabled with a permissions or required hardware level problem?

This is usually not a problem. The system detects the GPU during POST and it is only passed through to a VM after it completed booting. Passthrough of the first or only GPU can be a bit more tricky thern passthrough of additional GPUs, but that's about it.

 

[dabaers]

This is usually not a problem. The system detects the GPU during POST and it is only passed through to a VM after it completed booting. Passthrough of the first or only GPU can be a bit more tricky thern passthrough of additional GPUs, but that's about it.

Well then I am stuck, I have no clue what Im missing here, the only thing I can think of is that it's some kind of hardware issue but this is obviously not my area of expertise. Also, thank you for all the help so far!

 

[leesteken]

Can you please share your VM configuration file from /etc/pve/qemu-server/ and show the groups in a more useful manner using for d in /sys/kernel/iommu_groups/*/devices/*; do n=${d#*/iommu_groups/*}; n=${n%%/*}; printf 'IOMMU group %s ' "$n"; lspci -nn "${d##*/}"; done after removing the pcie_acs_override=downstream,multifunction (because that completely invalidates all information we can gather from the IOMMU groups)?

 

[dabaers]

Can you please share your VM configuration file from /etc/pve/qemu-server/ and show the groups in a more useful manner using for d in /sys/kernel/iommu_groups/*/devices/*; do n=${d#*/iommu_groups/*}; n=${n%%/*}; printf 'IOMMU group %s ' "$n"; lspci -nn "${d##*/}"; done after removing the pcie_acs_override=downstream,multifunction (because that completely invalidates all information we can gather from the IOMMU groups)?

So cleared the line pcie_acs_override=downstream,multifunction from grub and then used nano /etc/pve/qemu-server/ and the result came back saying this is a directory and the area is blank

 

[leesteken]

ls /etc/pve/qemu-server/ should show VM configuration files, ending in .conf and staring with the numeric VM-identifier. You can look at the configuration file with cat /etc/pve/qemu-server/100.conf for example for VM number 100.
Don't forget to run update-grub after making changes to /etc/default/grub and to reboot for the changes to take effect.

 

[dabaers]

ls /etc/pve/qemu-server/ should show VM configuration files, ending in .conf and staring with the numeric VM-identifier. You can look at the configuration file with cat /etc/pve/qemu-server/100.conf for example for VM number 100.
Don't forget to run update-grub after making changes to /etc/default/grub and to reboot for the changes to take effect.

I did run the update for grub, thanks for the reminder and here is the result!

root@Homestead:~# cat /etc/pve/qemu-server/103.conf
agent: 1
balloon: 1024
bios: ovmf
boot: order=scsi0;net0
cores: 4
cpu: host,hidden=1,flags=+pcid
machine: pc-q35-6.0
memory: 8192
name: Windows-Tester-1
net0: virtio=16:8A:36:9B:E5:42,bridge=vmbr0,firewall=1
numa: 1
ostype: win10
scsi0: Local-Proxmox:vm-103-disk-0,size=40G
scsihw: virtio-scsi-pci
smbios1: uuid=fa39f51a-154d-4c7c-8dab-2facda7ceb2b
sockets: 1
unused0: Local-Proxmox:vm-103-disk-1
vmgenid: 139d7b1d-b5ef-4cce-8cdb-09274a1ae519
vmstatestorage: TrueNas

 

[leesteken]

Sorry, but I don't see any hostpci entries. Is this the VM you want to use for passthrough?
Can you please share the complete output of for d in /sys/kernel/iommu_groups/*/devices/*; do n=${d#*/iommu_groups/*}; n=${n%%/*}; printf 'IOMMU group %s ' "$n"; lspci -nn "${d##*/}"; done (without pcie_acs_override)?