GPU Pass through problems

[jensie]

Dear all,

My purpose is to set up a Windows Server 2019 VM and using the NVIDIA GPU.

Proxmox 8.1 boots on a ZFS Mirror

My hardware
Motherboard Z590 PRO4
GPU NVIDIA Corporation TU116 [GeForce GTX 1650]

BIOS settings adjustments (listed), ensures that PVE doesn't use my NVIDIA GPU but the motherboard graphics card

• Primary Graphic Adapter set to onboard
• Enable VT-d
• Enable SR-IOV
• Set PEG {NUMBER} ASPM L0sL1
• Enable 4G decoding
• CAM disabled
• IGPU Enabled
• Disable CSM/Legacy Boot

Proxmox settings
in /etc/modprobe.d/kvm.conf

options kvm ignore_msrs=Y report_ignored_msrs=0

in /etc/default/grub

GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt initcall_blacklist=sysfb_init pcie_acs_override=downstream,multifunction"

in /etc/kernel/cmdline

root=ZFS=rpool/ROOT/pve-1 boot=zfs intel_iommu=on iommu=pt initcall_blacklist=sysfb_init pcie_acs_override=downstream,multifunction

update-grub
pve-efiboot-tool refresh
reboot

dmesg | grep -e DMAR -e IOMMU

results



in /etc/modules

vfio
vfio_iommu_type1
vfio_pci

update-initramfs -u -k all

dmesg | grep remapping

results



in /etc/modprobe.d/pve-blacklist.conf

blacklist nouveau
blacklist nvidia
blacklist nvidiafb
blacklist snd_hda_codec_hdmi
blacklist snd_hda_intel
blacklist snd_hda_codec
blacklist snd_hda_core
blacklist radeon
blacklist amdgpu

pvesh get /nodes/pve1/hardware/pci --pci-class-blacklist ""

returns



in /etc/modprobe.d/vfio-pci.conf

options vfio-pci ids=10de:2188 disable_vga=1

Final result is that I'm still getting the following yellow message : No IOMMU detected, please activate it.See Documentation for further information. in the proxmox UI.

I'm totally stuck and have been browsing the internet to find any kind of clue how to solve this, without luck. Can someone put me on the right track ?

 

[leesteken]

What is the output of cat /proc/cmdline (and does it contain intel_iommu=on? Is VT-d enabled in the motherboard BIOS? Does your motherboard and CPU both support VT-d?

 

[jensie]

Thank you for your quick reply.

- Output is

initrd=\EFI\proxmox\6.5.11-6-pve\initrd.img-6.5.11-6-pve #root=ZFS=rpool/ROOT/pve-1 boot=zfs

.

- It doesn't contain intel_iommu=on

- VT-d is enabled on the motherboard BIOS

- I assume that my motherboard supports VT-d as it's mentioned in the official manual https://download.asrock.com/Manual/Z590 Pro4.pdf
My CPU is an 11th Gen Intel(R) Core(TM) i9-11900K and based on https://ark.intel.com/content/www/u...1900k-processor-16m-cache-up-to-5-30-ghz.html, it supports Intel® Virtualization Technology for Directed I/O (VT-d)

 

[leesteken]

- Output is

initrd=\EFI\proxmox\6.5.11-6-pve\initrd.img-6.5.11-6-pve #root=ZFS=rpool/ROOT/pve-1 boot=zfs

.

Why is there a # before root=ZFS=rpool/ROOT/pve-1?

- It doesn't contain intel_iommu=on

You need to add intel_iommu=on to the kernel parameters for IOMMU to be active on Intel, as explained in the manual: https://pve.proxmox.com/pve-docs/pve-admin-guide.html#qm_pci_passthrough . Maybe you missed a step from the systemd-boot section? Note that your Proxmox probably does not use GRUB and please try without the pcie_acs_override first.

 

[jensie]

Why is there a # before root=ZFS=rpool/ROOT/pve-1?

You need to add intel_iommu=on to the kernel parameters for IOMMU to be active on Intel, as explained in the manual: https://pve.proxmox.com/pve-docs/pve-admin-guide.html#qm_pci_passthrough . Maybe you missed a step from the systemd-boot section? Note that your Proxmox probably does not use GRUB and please try without the pcie_acs_override first.

I removed the initial commented part of the file and got the following result

initrd=\EFI\proxmox\6.5.11-6-pve\initrd.img-6.5.11-6-pve root=ZFS=rpool/ROOT/pve-1 boot=zfs intel_iommu=on iommu=pt initcall_blacklist=sysfb_init pcie_acs_override=downstream,multifunction

Also if i'm looking now at

pvesh get /nodes/pve1/hardware/pci --pci-class-blacklist ""

I'm getting the different PCI devices grouped.

Now need to test with the VM if it is really working, but it feels like you found my mistake. Nogmaals bedankt voor je hulp @leesteken