Problem with GPU Passthrough in a Virtual Machine

[g3falsht]

Hello!
I'm having a problem with the video card being flashed into the virtual machine.

1. IOMMU is enabled, the video card is visible via LSPCI

0000:00:00.0 Host bridge: Intel Corporation Sky Lake-E DMI3 Registers (rev 07)
0000:00:04.0 System peripheral: Intel Corporation Sky Lake-E CBDMA Registers (rev 07)
0000:64:00.0 PCI bridge: Intel Corporation Sky Lake-E PCI Express Root Port 1A (rev 07)
0000:64:05.0 System peripheral: Intel Corporation Sky Lake-E VT-d (rev 07)
0000:64:05.2 System peripheral: Intel Corporation Sky Lake-E RAS Configuration Registers (rev 07)
0000:64:05.4 PIC: Intel Corporation Sky Lake-E IOxAPIC Configuration Registers (rev 07)
0000:64:08.0 System peripheral: Intel Corporation Sky Lake-E Integrated Memory Controller (rev 07)
0000:64:09.0 System peripheral: Intel Corporation Sky Lake-E Integrated Memory Controller (rev 07)
0000:64:0a.0 System peripheral: Intel Corporation Sky Lake-E Integrated Memory Controller (rev 07)
0000:64:0a.1 System peripheral: Intel Corporation Sky Lake-E Integrated Memory Controller (rev 07)
0000:64:0a.2 System peripheral: Intel Corporation Sky Lake-E Integrated Memory Controller (rev 07)
0000:64:0a.3 System peripheral: Intel Corporation Sky Lake-E Integrated Memory Controller (rev 07)
0000:64:0a.4 System peripheral: Intel Corporation Sky Lake-E Integrated Memory Controller (rev 07)
0000:64:0a.5 System peripheral: Intel Corporation Sky Lake-E LM Channel 1 (rev 07)
0000:64:0a.6 System peripheral: Intel Corporation Sky Lake-E LMS Channel 1 (rev 07)
0000:64:0a.7 System peripheral: Intel Corporation Sky Lake-E LMDP Channel 1 (rev 07)
0000:64:0b.0 System peripheral: Intel Corporation Sky Lake-E DECS Channel 2 (rev 07)
0000:64:0b.1 System peripheral: Intel Corporation Sky Lake-E LM Channel 2 (rev 07)
0000:64:0b.2 System peripheral: Intel Corporation Sky Lake-E LMS Channel 2 (rev 07)
0000:64:0b.3 System peripheral: Intel Corporation Sky Lake-E LMDP Channel 2 (rev 07)
0000:64:0c.0 System peripheral: Intel Corporation Sky Lake-E Integrated Memory Controller (rev 07)
0000:64:0c.1 System peripheral: Intel Corporation Sky Lake-E Integrated Memory Controller (rev 07)
0000:64:0c.2 System peripheral: Intel Corporation Sky Lake-E Integrated Memory Controller (rev 07)
0000:64:0c.3 System peripheral: Intel Corporation Sky Lake-E Integrated Memory Controller (rev 07)
0000:64:0c.4 System peripheral: Intel Corporation Sky Lake-E Integrated Memory Controller (rev 07)
0000:64:0c.5 System peripheral: Intel Corporation Sky Lake-E LM Channel 1 (rev 07)
0000:64:0c.6 System peripheral: Intel Corporation Sky Lake-E LMS Channel 1 (rev 07)
0000:64:0c.7 System peripheral: Intel Corporation Sky Lake-E LMDP Channel 1 (rev 07)
0000:64:0d.0 System peripheral: Intel Corporation Sky Lake-E DECS Channel 2 (rev 07)
0000:64:0d.1 System peripheral: Intel Corporation Sky Lake-E LM Channel 2 (rev 07)
0000:64:0d.2 System peripheral: Intel Corporation Sky Lake-E LMS Channel 2 (rev 07)
0000:64:0d.3 System peripheral: Intel Corporation Sky Lake-E LMDP Channel 2 (rev 07)
0000:65:00.0 PCI bridge: Intel Corporation Device 37c0 (rev 09)
0000:66:03.0 PCI bridge: Intel Corporation Device 37c5 (rev 09)
0000:67:00.0 Ethernet controller: Intel Corporation Ethernet Connection X722 for 1GbE (rev 09)
0000:67:00.1 Ethernet controller: Intel Corporation Ethernet Connection X722 for 1GbE (rev 09)
0000:67:00.2 Ethernet controller: Intel Corporation Ethernet Connection X722 for 1GbE (rev 09)
0000:67:00.3 Ethernet controller: Intel Corporation Ethernet Connection X722 for 1GbE (rev 09)
0000:b2:00.0 PCI bridge: Intel Corporation Sky Lake-E PCI Express Root Port 1A (rev 07)
0000:b2:05.0 System peripheral: Intel Corporation Sky Lake-E VT-d (rev 07)
0000:b2:05.2 System peripheral: Intel Corporation Sky Lake-E RAS Configuration Registers (rev 07)
0000:b2:05.4 PIC: Intel Corporation Sky Lake-E IOxAPIC Configuration Registers (rev 07)
0000:b2:05.5 RAID bus controller: Intel Corporation Device 201d (rev 07)
0000:b2:0e.0 Performance counters: Intel Corporation Sky Lake-E KTI 0 (rev 07)
0000:b2:0e.1 System peripheral: Intel Corporation Sky Lake-E UPI Registers (rev 07)
0000:b2:0f.0 Performance counters: Intel Corporation Sky Lake-E KTI 0 (rev 07)
0000:b2:0f.1 System peripheral: Intel Corporation Sky Lake-E UPI Registers (rev 07)
0000:b2:10.0 Performance counters: Intel Corporation Sky Lake-E KTI 0 (rev 07)
0000:b2:10.1 System peripheral: Intel Corporation Sky Lake-E UPI Registers (rev 07)
0000:b2:12.0 Performance counters: Intel Corporation Sky Lake-E M3KTI Registers (rev 07)
0000:b2:12.1 Performance counters: Intel Corporation Sky Lake-E M3KTI Registers (rev 07)
0000:b2:12.2 System peripheral: Intel Corporation Sky Lake-E M3KTI Registers (rev 07)
0000:b2:12.4 Performance counters: Intel Corporation Sky Lake-E M3KTI Registers (rev 07)
0000:b2:12.5 Performance counters: Intel Corporation Sky Lake-E M3KTI Registers (rev 07)
0000:b2:15.0 System peripheral: Intel Corporation Sky Lake-E M2PCI Registers (rev 07)
0000:b2:15.1 Performance counters: Intel Corporation Device 2088 (rev 07)
0000:b2:16.0 System peripheral: Intel Corporation Sky Lake-E M2PCI Registers (rev 07)
0000:b2:16.1 Performance counters: Intel Corporation Device 2088 (rev 07)
0000:b2:16.4 System peripheral: Intel Corporation Sky Lake-E M2PCI Registers (rev 07)
0000:b2:16.5 Performance counters: Intel Corporation Device 2088 (rev 07)
0000:b2:17.0 System peripheral: Intel Corporation Sky Lake-E M2PCI Registers (rev 07)
0000:b2:17.1 Performance counters: Intel Corporation Device 2088 (rev 07)
0000:b3:00.0 RAID bus controller: LSI Logic / Symbios Logic MegaRAID Tri-Mode SAS3508 (rev 01)
10000:00:00.0 PCI bridge: Intel Corporation Sky Lake-E PCI Express Root Port 1A (rev 07)
10000:00:01.0 PCI bridge: Intel Corporation Sky Lake-E PCI Express Root Port 1B (rev 07)
10000:00:02.0 PCI bridge: Intel Corporation Sky Lake-E PCI Express Root Port 1C (rev 07)
10000:00:03.0 PCI bridge: Intel Corporation Sky Lake-E PCI Express Root Port 1D (rev 07)
10000:01:00.0 VGA compatible controller: NVIDIA Corporation TU104GL [Quadro RTX 4000] (rev a1)
10000:01:00.1 Audio device: NVIDIA Corporation Device 10f8 (rev a1)
10000:01:00.2 USB controller: NVIDIA Corporation Device 1ad8 (rev a1)
10000:01:00.3 Serial bus controller [0c80]: NVIDIA Corporation Device 1ad9 (rev a1)
10001:00:02.0 PCI bridge: Intel Corporation Sky Lake-E PCI Express Root Port 1C (rev 07)
10001:00:03.0 PCI bridge: Intel Corporation Sky Lake-E PCI Express Root Port 1D (rev 07)
10001:01:00.0 Non-Volatile memory controller: Intel Corporation NVMe Datacenter SSD [3DNAND, Beta Rock Controller]
10001:02:00.0 Non-Volatile memory controller: Intel Corporation NVMe Datacenter SSD [3DNAND, Beta Rock Controller]

As you can see, the video card is there and displayed in LSPCI, but it is not visible in the web interface.

10000:01:00.0 VGA compatible controller: NVIDIA Corporation TU104GL [Quadro RTX 4000] (rev a1)
10000:01:00.1 Audio device: NVIDIA Corporation Device 10f8 (rev a1)
10000:01:00.2 USB controller: NVIDIA Corporation Device 1ad8 (rev a1)

Here are some screenshots from the Proxmox web interface




As you can see the NVIDIA graphics card is not in the list of PCI device choices in the web interface.
The settings are all correct, this is not the first time we do this kind of video forwarding task.
The server configuration is attached below.

CPU(s) 32 x Intel(R) Xeon(R) Gold 6246R CPU @ 3.40GHz (1 Socket)

Kernel Version Linux 5.4.106-1-pve #1 SMP PVE 5.4.106-1 (Fri, 19 Mar 2021 11:08:47 +0100)

PVE Manager Version pve-manager/6.4-13/9f411e79

Intel Xeon Gold 6246R/ Intel® Server System R2208WF0ZSR 2U
/ DDR4 32GB RDIMM (PC4-23400) 2933MHz ECC Reg 1.2V x6

Thank you for your answers!

 

[dcsapak]

try to set it via the cli:

qm set ID -hostpciX '10000:01:00'

 

[g3falsht]

try to set it via the cli:

qm set ID -hostpciX '10000:01:00'

 

[dcsapak]

hi,
yes, unfortunately, our code only allows 4 character pci domains (e.g. 0000). i sent patches [0][1] to the devel list so that longer
domains are allowed too, but this will probably not be backported to pve 6.4

0: https://lists.proxmox.com/pipermail/pve-devel/2022-February/051872.html
1: https://lists.proxmox.com/pipermail/pve-devel/2022-February/051873.html

 

[g3falsht]

hi,
yes, unfortunately, our code only allows 4 character pci domains (e.g. 0000). i sent patches [0][1] to the devel list so that longer
domains are allowed too, but this will probably not be backported to pve 6.4

0: https://lists.proxmox.com/pipermail/pve-devel/2022-February/051872.html
1: https://lists.proxmox.com/pipermail/pve-devel/2022-February/051873.html

Thanks for the reply!
Do we understand correctly that we will not be able to route this video card to Proxmox?
What solution is possible in this case?

Update Proxmox after the release of these patches?

 

[dcsapak]

Do we understand correctly that we will not be able to route this video card to Proxmox?
What solution is possible in this case?

as a temporary workaround, you could check how we add the pci device to the vm (with 'qm showcmd ID --pretty') and add that to the 'args' config
this will be added to the qemu commandline, but is then not managed by pve (so you don't see it in the webui, the additional checks, like migration blocker, are not done etc.)

Update Proxmox after the release of these patches?

thats the safest route, yes

 

[g3falsht]

as a temporary workaround, you could check how we add the pci device to the vm (with 'qm showcmd ID --pretty') and add that to the 'args' config
this will be added to the qemu commandline, but is then not managed by pve (so you don't see it in the webui, the additional checks, like migration blocker, are not done etc.)


thats the safest route, yes

I tried to edit the virtual machine configuration file and put the PCI line in there. Unfortunately this did not help.

What is the approximate release date of these patches?