I just got this fully working; thought I'd share my notes on what it required to succeed in my context.
Host: HP Z2 SFF box, Xeon CPU with full virtualization support.
A. MUST follow standard ProxMox host pass-through instructions (.../pve-docs/chapter-qm.html#qm_pci_passthrough) including:
1) In /etc/kernel/cmdline, MUST add
2) Must incorporate the vfio drivers
3) Must find the HW ID's for GPU cards and vendors, incorporate into a file in /etc/modules.d -- see below for unusual bit
4) Do the same for blocking the usual (nouveau) driver
5) Redo kernel etc with
6) Ensure your GPU is in an isolated IOMMU group (If needed, great insights here: https://vfio.blogspot.com/2014/08/iommu-groups-inside-and-out.html
IMPORTANT: For the T1000, pass through both the GPU and the Audio device. You don't need to use the audio, but the nVidia driver will not recognize the GPU if it doesn't see the audio!
B. Set up a typical Win10 VM following best practices (eg https://pve.proxmox.com/wiki/Windows_10_guest_best_practices )
IMPORTANT: For the T1000, the PCIe device must be defined including both hardware devices, and PCIe set. Your PCI slot may be different but the *.conf line on mine is:
NOTE that I removed the ".0" at the end, to include both the .0 GPU and the .1 Audio.
C. Download the driver in the VM, once you have Windows set up. I used
NOTES:
HINT: If the driver installs but the utilities fail to work, check whether you checked the "pcie" box in the ProxMox hardware settings.
Hope that helps!
Host: HP Z2 SFF box, Xeon CPU with full virtualization support.
A. MUST follow standard ProxMox host pass-through instructions (.../pve-docs/chapter-qm.html#qm_pci_passthrough) including:
1) In /etc/kernel/cmdline, MUST add
intel_ iommu=on and SHOULD add iommu=pt (the latter improves performance if supported. Direct pass-thru)2) Must incorporate the vfio drivers
3) Must find the HW ID's for GPU cards and vendors, incorporate into a file in /etc/modules.d -- see below for unusual bit
4) Do the same for blocking the usual (nouveau) driver
5) Redo kernel etc with
proxmox-boot-tool refresh and update-initramfs -u -k all then reboot6) Ensure your GPU is in an isolated IOMMU group (If needed, great insights here: https://vfio.blogspot.com/2014/08/iommu-groups-inside-and-out.html
IMPORTANT: For the T1000, pass through both the GPU and the Audio device. You don't need to use the audio, but the nVidia driver will not recognize the GPU if it doesn't see the audio!
B. Set up a typical Win10 VM following best practices (eg https://pve.proxmox.com/wiki/Windows_10_guest_best_practices )
IMPORTANT: For the T1000, the PCIe device must be defined including both hardware devices, and PCIe set. Your PCI slot may be different but the *.conf line on mine is:
Code:
hostpci0: 0000:01:00,pcie=1C. Download the driver in the VM, once you have Windows set up. I used
https://us.download.nvidia.com/Windows/Quadro_Certified/551.61/551.61-quadro-rtx-desktop-notebook-win10-win11-64bit-international-dch-whql.exeNOTES:
- Once all of the above is correct, the driver will recognize your card and be willing to install.
- No need to attach a display
- Even though you MUST pass through Audio for the card to be seen by the driver, it is not necessary to install the audio driver nor the RTX desktop manager (although you certainly can if you like)
- The display/driver configurator will be in your system tray
- The graphics driver alone will enable use of CUDA by your applications
- A nice test of the install, and of performance, is NeatVideo's NeatBench (available for Win, Mac, Linux) here: https://www.neatvideo.com/download/neatbench
HINT: If the driver installs but the utilities fail to work, check whether you checked the "pcie" box in the ProxMox hardware settings.
Hope that helps!
Last edited: