GPU Passthrough crash PVE

Code: 
class    device id           iommugroup vendor device_name                                                                             mdev subsystem_device subsystem_device_name                subsystem_vendor subsystem_vendor_name    vendor_name
0x010601 0xa352 0000:00:17.0          8 0x8086 Cannon Lake PCH SATA AHCI Controller                                                         0x1a1d                                                0x15d9           Super Micro Computer Inc Intel Corporation
0x010601 0x1166 0000:0c:00.0         25 0x1b21 ASM1166 Serial ATA Controller                                                                0x2116                                                0x2116           ZyDAS Technology Corp.   ASMedia Technology Inc.
0x020000 0x15bb 0000:00:1f.6         17 0x8086 Ethernet Connection (7) I219-LM                                                              0x15bb                                                0x15d9           Super Micro Computer Inc Intel Corporation
0x020000 0x10fb 0000:01:00.0          2 0x8086 82599ES 10-Gigabit SFI/SFP+ Network Connection                                               0x17d3           Ethernet 10Gb 2-port 560SFP+ Adapter 0x103c           Hewlett-Packard Company  Intel Corporation
0x020000 0x10fb 0000:01:00.1          2 0x8086 82599ES 10-Gigabit SFI/SFP+ Network Connection                                               0x17d3           Ethernet 10Gb 2-port 560SFP+ Adapter 0x103c           Hewlett-Packard Company  Intel Corporation
0x020000 0x1521 0000:04:00.0         18 0x8086 I350 Gigabit Network Connection                                                              0x00a1           Ethernet Server Adapter I350-T4      0x8086           Intel Corporation        Intel Corporation
0x020000 0x1521 0000:04:00.1         19 0x8086 I350 Gigabit Network Connection                                                              0x00a1           Ethernet Server Adapter I350-T4      0x8086           Intel Corporation        Intel Corporation
0x020000 0x1521 0000:04:00.2         20 0x8086 I350 Gigabit Network Connection                                                              0x00a1           Ethernet Server Adapter I350-T4      0x8086           Intel Corporation        Intel Corporation
0x020000 0x1521 0000:04:00.3         21 0x8086 I350 Gigabit Network Connection                                                              0x00a1           Ethernet Server Adapter I350-T4      0x8086           Intel Corporation        Intel Corporation
0x020000 0x1533 0000:07:00.0         22 0x8086 I210 Gigabit Network Connection                                                              0x1533                                                0x15d9           Super Micro Computer Inc Intel Corporation
0x030000 0x3e96 0000:00:02.0          0 0x8086 CoffeeLake-S GT2 [UHD Graphics P630]                                                         0x1a1d                                                0x15d9           Super Micro Computer Inc Intel Corporation
0x040300 0xa348 0000:00:1f.3         17 0x8086 Cannon Lake PCH cAVS                                                                         0x1a1d                                                0x15d9           Super Micro Computer Inc Intel Corporation
0x050000 0xa36f 0000:00:14.2          5 0x8086 Cannon Lake PCH Shared SRAM                                                                  0x7270                                                0x8086           Intel Corporation        Intel Corporation
0x060000 0x3ec6 0000:00:00.0          1 0x8086 8th Gen Core Processor Host Bridge/DRAM Registers                                            0x1a1d                                                0x15d9           Super Micro Computer Inc Intel Corporation
0x060100 0xa309 0000:00:1f.0         17 0x8086 C246 Chipset LPC/eSPI Controller                                                             0x1a1d                                                0x15d9           Super Micro Computer Inc Intel Corporation
0x060400 0x1901 0000:00:01.0          2 0x8086 6th-10th Gen Core Processor PCIe Controller (x16)                                            0x1a1d                                                0x15d9           Super Micro Computer Inc Intel Corporation
0x060400 0xa340 0000:00:1b.0          9 0x8086 Cannon Lake PCH PCI Express Root Port #17                                                    0x1a1d                                                0x15d9           Super Micro Computer Inc Intel Corporation
0x060400 0xa32c 0000:00:1b.4         10 0x8086 Cannon Lake PCH PCI Express Root Port #21                                                    0x7270                                                0x8086           Intel Corporation        Intel Corporation
0x060400 0xa338 0000:00:1c.0         11 0x8086 Cannon Lake PCH PCI Express Root Port #1                                                     0x1a1d                                                0x15d9           Super Micro Computer Inc Intel Corporation
0x060400 0xa33d 0000:00:1c.5         12 0x8086 Cannon Lake PCH PCI Express Root Port #6                                                     0x1a1d                                                0x15d9           Super Micro Computer Inc Intel Corporation
0x060400 0xa33e 0000:00:1c.6         13 0x8086 Cannon Lake PCH PCI Express Root Port #7                                                     0x1a1d                                                0x15d9           Super Micro Computer Inc Intel Corporation
0x060400 0xa33f 0000:00:1c.7         14 0x8086 Cannon Lake PCH PCI Express Root Port #8                                                     0x1a1d                                                0x15d9           Super Micro Computer Inc Intel Corporation
0x060400 0xa330 0000:00:1d.0         15 0x8086 Cannon Lake PCH PCI Express Root Port #9                                                     0x1a1d                                                0x15d9           Super Micro Computer Inc Intel Corporation
0x060400 0x1150 0000:08:00.0         23 0x1a03 AST1150 PCI-to-PCI Bridge                                                                    0x1150                                                0x1a03           ASPEED Technology, Inc.  ASPEED Technology, Inc.
0x060401 0x8113 0000:0a:00.0         24 0x10e3 89HPEB383 PCIe-to-PCI Bridge                                                                 0x1a1d                                                0x15d9           Super Micro Computer Inc Tundra Semiconductor Corp.
0x078000 0xa360 0000:00:16.0          7 0x8086 Cannon Lake PCH HECI Controller                                                              0x1a1d                                                0x15d9           Super Micro Computer Inc Intel Corporation
0x078000 0xa328 0000:00:1e.0         16 0x8086 Cannon Lake PCH Serial IO UART Host Controller                                               0x1a1d                                                0x15d9           Super Micro Computer Inc Intel Corporation
0x088000 0x1911 0000:00:08.0          3 0x8086 Xeon E3-1200 v5/v6 / E3-1500 v5 / 6th/7th/8th Gen Core Processor Gaussian Mixture Model      0x1a1d                                                0x15d9           Super Micro Computer Inc Intel Corporation
0x0c0330 0xa36d 0000:00:14.0          5 0x8086 Cannon Lake PCH USB 3.1 xHCI Host Controller                                                 0x1a1d                                                0x15d9           Super Micro Computer Inc Intel Corporation
0x0c0500 0xa323 0000:00:1f.4         17 0x8086 Cannon Lake PCH SMBus Controller                                                             0x1a1d                                                0x15d9           Super Micro Computer Inc Intel Corporation
0x0c8000 0xa368 0000:00:15.0          6 0x8086 Cannon Lake PCH Serial IO I2C Controller #0                                                  0x1a1d                                                0x15d9           Super Micro Computer Inc Intel Corporation
0x0c8000 0xa369 0000:00:15.1          6 0x8086 Cannon Lake PCH Serial IO I2C Controller #1                                                  0x1a1d                                                0x15d9           Super Micro Computer Inc Intel Corporation
0x0c8000 0xa324 0000:00:1f.5         17 0x8086 Cannon Lake PCH SPI Controller                                                               0x1a1d                                                0x15d9           Super Micro Computer Inc Intel Corporation
0x118000 0xa379 0000:00:12.0          4 0x8086 Cannon Lake PCH Thermal Controller                                                           0x1a1d                                                0x15d9           Super Micro Computer Inc Intel Corporation
Sorry for the late reply. For your information, currently I have no graphics card installed since it is not working with two devices in the two PCIe slots. I maybe should have mentioned that
 
spetrillo said:
Any chance to put the cards back and then run the previous command? I want to see where the cards land in the IOMMU group.
Click to expand...
Hey there spetrillo. I put in a second 10G network card in the test and as expected now the 10G card in the top x16 slot and the x16 slot below share the same IOMMU group now. Any idea what I can do here? Maybe there is a Bios setting or did Supermicro just screw up the IOMMU grouping? I know from a guy with an ASUS WS C246 that his groups are seperate for passthrough.

All of this would not even bother me if I got the iGPU to work in my Windows 10 VM. I can't get a display out, probably because of the ASPEED BMC sharing the ports. I have no hardware error inside Device Manager (like Error 43) but once I startup the VM my display stays black. On a Ubuntu VM I just passthrough the iGPU and it works right away. No config mods necessary etc. The Windows 10 VM worked with a GTX 970 as long as just one PCIe x16 slot is occupied. So I am stuck in a bind because I need my 10G and 1G network cards and I can't run them in addition to a GTX970.

Code: 
class    device id           iommugroup vendor device_name                                                                             mdev subsystem_device subsystem_devi
0x010601 0xa352 0000:00:17.0          8 0x8086 Cannon Lake PCH SATA AHCI Controller                                                         0x1a1d                        
0x010601 0x1166 0000:0e:00.0         25 0x1b21 ASM1166 Serial ATA Controller                                                                0x2116                        
0x020000 0x15bb 0000:00:1f.6         17 0x8086 Ethernet Connection (7) I219-LM                                                              0x15bb                        
0x020000 0x10fb 0000:01:00.0          2 0x8086 82599ES 10-Gigabit SFI/SFP+ Network Connection                                               0x17d3           Ethernet 10Gb
0x020000 0x10fb 0000:01:00.1          2 0x8086 82599ES 10-Gigabit SFI/SFP+ Network Connection                                               0x17d3           Ethernet 10Gb
0x020000 0x1557 0000:03:00.0          2 0x8086 82599 10 Gigabit Network Connection                                                          0x0317                        
0x020000 0x1521 0000:06:00.0         18 0x8086 I350 Gigabit Network Connection                                                              0x00a1           Ethernet Serve
0x020000 0x1521 0000:06:00.1         19 0x8086 I350 Gigabit Network Connection                                                              0x00a1           Ethernet Serve
0x020000 0x1521 0000:06:00.2         20 0x8086 I350 Gigabit Network Connection                                                              0x00a1           Ethernet Serve
0x020000 0x1521 0000:06:00.3         21 0x8086 I350 Gigabit Network Connection                                                              0x00a1           Ethernet Serve
0x020000 0x1533 0000:09:00.0         22 0x8086 I210 Gigabit Network Connection                                                              0x1533                        
0x030000 0x3e96 0000:00:02.0          0 0x8086 CoffeeLake-S GT2 [UHD Graphics P630]                                                         0x1a1d                        
0x040300 0xa348 0000:00:1f.3         17 0x8086 Cannon Lake PCH cAVS                                                                         0x1a1d                        
0x050000 0xa36f 0000:00:14.2          5 0x8086 Cannon Lake PCH Shared SRAM                                                                  0x7270                        
0x060000 0x3ec6 0000:00:00.0          1 0x8086 8th Gen Core Processor Host Bridge/DRAM Registers                                            0x1a1d                        
0x060100 0xa309 0000:00:1f.0         17 0x8086 C246 Chipset LPC/eSPI Controller                                                             0x1a1d                        
0x060400 0x1901 0000:00:01.0          2 0x8086 6th-10th Gen Core Processor PCIe Controller (x16)                                            0x1a1d                        
0x060400 0x1905 0000:00:01.1          2 0x8086 Xeon E3-1200 v5/E3-1500 v5/6th Gen Core Processor PCIe Controller (x8)                       0x1a1d                        
0x060400 0xa340 0000:00:1b.0          9 0x8086 Cannon Lake PCH PCI Express Root Port #17                                                    0x1a1d                        
0x060400 0xa32c 0000:00:1b.4         10 0x8086 Cannon Lake PCH PCI Express Root Port #21                                                    0x7270                        
0x060400 0xa338 0000:00:1c.0         11 0x8086 Cannon Lake PCH PCI Express Root Port #1                                                     0x1a1d                        
0x060400 0xa33d 0000:00:1c.5         12 0x8086 Cannon Lake PCH PCI Express Root Port #6                                                     0x1a1d                        
0x060400 0xa33e 0000:00:1c.6         13 0x8086 Cannon Lake PCH PCI Express Root Port #7                                                     0x1a1d                        
0x060400 0xa33f 0000:00:1c.7         14 0x8086 Cannon Lake PCH PCI Express Root Port #8                                                     0x1a1d                        
0x060400 0xa330 0000:00:1d.0         15 0x8086 Cannon Lake PCH PCI Express Root Port #9                                                     0x1a1d                        
0x060400 0x1150 0000:0a:00.0         23 0x1a03 AST1150 PCI-to-PCI Bridge                                                                    0x1150                        
0x060401 0x8113 0000:0c:00.0         24 0x10e3 89HPEB383 PCIe-to-PCI Bridge                                                                 0x1a1d                        
0x078000 0xa360 0000:00:16.0          7 0x8086 Cannon Lake PCH HECI Controller                                                              0x1a1d                        
0x078000 0xa328 0000:00:1e.0         16 0x8086 Cannon Lake PCH Serial IO UART Host Controller                                               0x1a1d                        
0x088000 0x1911 0000:00:08.0          3 0x8086 Xeon E3-1200 v5/v6 / E3-1500 v5 / 6th/7th/8th Gen Core Processor Gaussian Mixture Model      0x1a1d                        
0x0c0330 0xa36d 0000:00:14.0          5 0x8086 Cannon Lake PCH USB 3.1 xHCI Host Controller                                                 0x1a1d                        
0x0c0500 0xa323 0000:00:1f.4         17 0x8086 Cannon Lake PCH SMBus Controller                                                             0x1a1d                        
0x0c8000 0xa368 0000:00:15.0          6 0x8086 Cannon Lake PCH Serial IO I2C Controller #0                                                  0x1a1d                        
0x0c8000 0xa369 0000:00:15.1          6 0x8086 Cannon Lake PCH Serial IO I2C Controller #1                                                  0x1a1d                        
0x0c8000 0xa324 0000:00:1f.5         17 0x8086 Cannon Lake PCH SPI Controller                                                               0x1a1d                        
0x118000 0xa379 0000:00:12.0          4 0x8086 Cannon Lake PCH Thermal Controller                                                           0x1a1d
 
Last edited:
Hold on...I am glad you got what I expected you would get!

Now please edit /etc/kernel/cmdline and add pcie_acs_override=downstream,multifunction to the end of the cmd string. Then I need you to run update-initramfs -u -k all and rebuild the kernel parameters. Reboot and then run pvesh get /nodes/{nodename}/hardware/pci --pci-class-blacklist "". When you get the results you should see both cards in their own IOMMU group.
 
Hey spetrillo. I know about acs override. But isn't that a security risk since the VMs are not isolated anymore from each other? I wanted to avoid that. I was hoping for a different solution.

I also found this reddit thread because I remember issues with ACS on those particular boards
https://www.reddit.com/r/VFIO/comments/97da1s/intel_c246_suspect_300_series_acs_quirk_breaks/

But not sure what Kernel they are talking about. Apparently older ones work fine with passthrough etc.
 
Last edited:
Its the way your mboard processes the IOMMU devices. Sometimes you gotta force it

The decision is up to you but its the only way I was able to get things the way I wanted.
 
You must log in or register to reply here.
Top Bottom