[SOLVED] Issues passing through GPU to Remote Gaming VM

[gsmitty]

Hello!

I've been following this guide from Craft Computing on Youtube to create a Bazzite gaming VM that I can access remotely.

He includes a link to this cheatsheet for passing through a GPU to the Bazzite VM.

I've followed the guide meticulously for my Dell RTX 3070, but I get this error when I try to start the VM:


This is the output of

dmesg | grep -e IOMMU

:
**This forum post validated that output as correct/expected after using the recommendation for the /etc/default/grub config file.

This is the output of

dmesg | grep -i vfio

:


I'm at a loss as to how to fix this. I've verified that I'm using grub to boot and Intel Virtualization capabilities are enabled in the BIOS.

This is a new build on a Dell Precision 7820 with Dual Xeon 6146 Gold CPUs and I'm trying to pass through an NVIDIA RTX 3070.

Any input on where to look next would be appreciated!

 

[leesteken]

View attachment 87301

Assuming you are running a recent PVE (with Linux kernel 6.8 or higher, which you can check with cat /proc/cmdline), VT-d is not enabled in the motherboard BIOS. The CPU supports VT-d but I don't know the motherboard, but since it is a server I expect it to support it. Please double check the BIOS settings and enable VT-d. It is often confused with VT-x, which is for running VMs but you need VT-d also for IOMMU. What is the output of journalctl -b 0 | grep -i iommu?

 

[gsmitty]

Ok I'll verify.

Here is the output of that command:

Jun 19 13:13:41 pve kernel: Command line: BOOT_IMAGE=/vmlinuz-6.8.12-11-pve root=ZFS=/ROOT/pve-1 ro root=ZFS=rpool/ROOT/pve-1 boot=zfs quiet amd_iommu=on iommu=pt nomodeset pcie_acs_override=downstream initcall_blacklist=sysfb_init
Jun 19 13:13:41 pve kernel: Warning: PCIe ACS overrides enabled; This may allow non-IOMMU protected peer-to-peer DMA
Jun 19 13:13:41 pve kernel: Kernel command line: BOOT_IMAGE=/vmlinuz-6.8.12-11-pve root=ZFS=/ROOT/pve-1 ro root=ZFS=rpool/ROOT/pve-1 boot=zfs quiet amd_iommu=on iommu=pt nomodeset pcie_acs_override=downstream initcall_blacklist=sysfb_init
Jun 19 13:13:41 pve kernel: iommu: Default domain type: Passthrough (set via kernel command line)
Jun 19 13:13:48 pve pve-guests[2750]: cannot prepare PCI pass-through, IOMMU not present
Jun 19 13:13:48 pve pvesh[2744]: Starting VM 100 failed: cannot prepare PCI pass-through, IOMMU not present

 

[leesteken]

Ok I'll verify.

Here is the output of that command:

Jun 19 13:13:41 pve kernel: Command line: BOOT_IMAGE=/vmlinuz-6.8.12-11-pve root=ZFS=/ROOT/pve-1 ro root=ZFS=rpool/ROOT/pve-1 boot=zfs quiet amd_iommu=on iommu=pt nomodeset pcie_acs_override=downstream initcall_blacklist=sysfb_init

I hope you realize that amd_iommu=on if not only invalid and unnecessary but also makes no sense on an Intel platform.

Jun 19 13:13:41 pve kernel: Warning: PCIe ACS overrides enabled; This may allow non-IOMMU protected peer-to-peer DMA
Jun 19 13:13:41 pve kernel: Kernel command line: BOOT_IMAGE=/vmlinuz-6.8.12-11-pve root=ZFS=/ROOT/pve-1 ro root=ZFS=rpool/ROOT/pve-1 boot=zfs quiet amd_iommu=on iommu=pt nomodeset pcie_acs_override=downstream initcall_blacklist=sysfb_init
Jun 19 13:13:41 pve kernel: iommu: Default domain type: Passthrough (set via kernel command line)
Jun 19 13:13:48 pve pve-guests[2750]: cannot prepare PCI pass-through, IOMMU not present
Jun 19 13:13:48 pve pvesh[2744]: Starting VM 100 failed: cannot prepare PCI pass-through, IOMMU not present

It does indeed look like VT-d is not enabled in the motherboard BIOS and therefore IOMMU is not available.Please consult your motherboard manual and/or the system/motherboard manufacturer support to enable VT-d in the BIOS.

 

[gsmitty]

Ok, I didn't even notice that AMD was in that string. I'll get rid of it.

And thanks! I'll see what I need to do to enable.

 

[gsmitty]

Ok I corrected the immou setting and made sure VT-d was on.
In my case VT-d is enabled, but its not called that on this particular motherboard. I found the analogue and made sure it was enabled.

I'm still having the same issue

It seems to me the issue is with VFIO modules from these errors. I just don't know what they mean:

 

[leesteken]

Ok I corrected the immou setting and made sure VT-d was on.
In my case VT-d is enabled, but its not called that on this particular motherboard. I found the analogue and made sure it was enabled.

Please share how it was called (for others to learn and search for). Also, please share the output of journalctl -b 0 | grep -i iommu (in CODE-tags). Unless we see that outpu7t and certain specific lines in that, IOMMU might not be fully enabled or, depending on the PCIe layout and multiplexer of the motherboard, it might not support the necessary features for passthrough.

It seems to me the issue is with VFIO modules from these errors. I just don't know what they mean:
View attachment 87311

I also don't know but it would be much easier to search for if I could copy-paste (parts of) the error messages. Please use text and CODE-tags.

 

[gsmitty]

Ok the VT-d equivalent on a Dell Precision 7820 is called VT for Direct I/O. Its actually enabled by default.

And apologies on not putting it in code tags I'll give that below:

Output of journalctl -b 0 | grep -i iommu after I tried to boot the VM with the PCI device:

Jun 19 16:30:59 pve kernel: Command line: BOOT_IMAGE=/vmlinuz-6.8.12-11-pve root=ZFS=/ROOT/pve-1 ro root=ZFS=rpool/ROOT/pve-1 boot=zfs quiet intel_iommu=on
Jun 19 16:30:59 pve kernel: Kernel command line: BOOT_IMAGE=/vmlinuz-6.8.12-11-pve root=ZFS=/ROOT/pve-1 ro root=ZFS=rpool/ROOT/pve-1 boot=zfs quiet intel_iommu=on
Jun 19 16:30:59 pve kernel: DMAR: IOMMU enabled
Jun 19 16:30:59 pve kernel: iommu: Default domain type: Translated
Jun 19 16:30:59 pve kernel: iommu: DMA domain TLB invalidation policy: lazy mode
Jun 19 16:31:31 pve pvedaemon[2898]: cannot prepare PCI pass-through, IOMMU not present
Jun 19 16:31:31 pve pvedaemon[2684]: <root@pam> end task UPID:pve:00000B52:0000185F:68548FC3:qmstart:100:root@pam: cannot prepare PCI pass-through, IOMMU not present

Output of dmesg | grep -i vfio:

[   30.004549] VFIO - User Level meta-driver version: 0.3
[   30.011671] vfio-pci 0000:18:00.0: vgaarb: deactivate vga console
[   30.012653] vfio-pci 0000:18:00.0: vgaarb: VGA decodes changed: olddecodes=io+mem,decodes=none:owns=io+mem
[   30.012680] vfio-pci: probe of 0000:18:00.0 failed with error -22
[   30.012729] vfio_pci: add [10de:2484[ffffffff:ffffffff]] class 0x000000/00000000
[   30.012795] vfio-pci 0000:18:00.0: vgaarb: deactivate vga console
[   30.012810] vfio-pci 0000:18:00.0: vgaarb: VGA decodes changed: olddecodes=none,decodes=none:owns=io+mem
[   30.012823] vfio-pci: probe of 0000:18:00.0 failed with error -22
[   30.012883] vfio-pci: probe of 0000:18:00.1 failed with error -22
[   30.012905] vfio_pci: add [10de:228b[ffffffff:ffffffff]] class 0x000000/00000000

 

[leesteken]

Ok the VT-d equivalent on a Dell Precision 7820 is called VT for Direct I/O. Its actually enabled by default.

That make sense, Intel calls it "Intel® Virtualization Technology for Directed I/O (VT-d)": https://www.intel.com/content/www/u...sor-24-75m-cache-3-20-ghz/specifications.html

Output of journalctl -b 0 | grep -i iommu after I tried to boot the VM with the PCI device:

Jun 19 16:30:59 pve kernel: Command line: BOOT_IMAGE=/vmlinuz-6.8.12-11-pve root=ZFS=/ROOT/pve-1 ro root=ZFS=rpool/ROOT/pve-1 boot=zfs quiet intel_iommu=on
Jun 19 16:30:59 pve kernel: Kernel command line: BOOT_IMAGE=/vmlinuz-6.8.12-11-pve root=ZFS=/ROOT/pve-1 ro root=ZFS=rpool/ROOT/pve-1 boot=zfs quiet intel_iommu=on
Jun 19 16:30:59 pve kernel: DMAR: IOMMU enabled
Jun 19 16:30:59 pve kernel: iommu: Default domain type: Translated
Jun 19 16:30:59 pve kernel: iommu: DMA domain TLB invalidation policy: lazy mode
Jun 19 16:31:31 pve pvedaemon[2898]: cannot prepare PCI pass-through, IOMMU not present
Jun 19 16:31:31 pve pvedaemon[2684]: <root@pam> end task UPID:pve:00000B52:0000185F:68548FC3:qmstart:100:root@pam: cannot prepare PCI pass-through, IOMMU not present

intel_iommu=on is not necessary (since kernel 6.8): https://pve.proxmox.com/pve-docs/pve-admin-guide.html#qm_pci_passthrough
DMAR: IOMMU enabled looks good, but there are no lines about putting various devices into IOMMU groups, which is bad. That's probably why PVE claims cannot prepare PCI pass-through, IOMMU not present. Somehow IOMMU is enabled but the PCI(e) device are not put into IOMMU groups, which prevents PCI(e) passthrough. I don't know why; please ask Dell support.

 

[gsmitty]

Output of pvesh get /nodes/pve/hardware/pci --pci-class-blacklist "" (which the link you sent says is the way to test if I have IOMMU groups):

class    │ device │ id            │ iommugroup │ vendor │ device_name                                         
╞══════════╪════════╪═══════════════╪════════════╪════════╪══════════════════════════════════════════════════════
│ 0x010185 │ 0xa1bc │ 0000:00:16.2  │         -1 │ 0x8086 │ C620 Series Chipset Family IDE Redirection           
├──────────┼────────┼───────────────┼────────────┼────────┼──────────────────────────────────────────────────────
│ 0x010400 │ 0x201d │ 0000:17:05.5  │         -1 │ 0x8086 │ Volume Management Device NVMe RAID Controller       
├──────────┼────────┼───────────────┼────────────┼────────┼──────────────────────────────────────────────────────
│ 0x010601 │ 0xa182 │ 0000:00:17.0  │         -1 │ 0x8086 │ C620 Series Chipset Family SATA Controller [AHCI mode
├──────────┼────────┼───────────────┼────────────┼────────┼──────────────────────────────────────────────────────
│ 0x010802 │ 0x5006 │ 0000:5b:00.0  │         -1 │ 0x15b7 │ WD Black SN750 / PC SN730 NVMe SSD                   
├──────────┼────────┼───────────────┼────────────┼────────┼──────────────────────────────────────────────────────
│ 0x010802 │ 0x5016 │ 0000:5c:00.0  │         -1 │ 0x2646 │                                                     
├──────────┼────────┼───────────────┼────────────┼────────┼──────────────────────────────────────────────────────
│ 0x010802 │ 0x5016 │ 0000:5d:00.0  │         -1 │ 0x2646 │                                                     
├──────────┼────────┼───────────────┼────────────┼────────┼──────────────────────────────────────────────────────
│ 0x010802 │ 0x0116 │ 10000:01:00.0 │         -1 │ 0x1179 │ XG5 NVMe SSD Controller                             
├──────────┼────────┼───────────────┼────────────┼────────┼──────────────────────────────────────────────────────
│ 0x020000 │ 0x8125 │ 0000:02:00.0  │         -1 │ 0x10ec │ RTL8125 2.5GbE Controller                           
├──────────┼────────┼───────────────┼────────────┼────────┼──────────────────────────────────────────────────────
│ 0x030000 │ 0x2484 │ 0000:18:00.0  │         -1 │ 0x10de │ GA104 [GeForce RTX 3070]                             
├──────────┼────────┼───────────────┼────────────┼────────┼──────────────────────────────────────────────────────
│

The list is longer, but doesn't this indicate that I have IOMMU groups? And I can see my graphics card on this list.

 

[gsmitty]

I ended up switching the slot of my card and changing to UEFI boot instead of using GRUB. That fixed it an basically invalidates this whole thread. Marking as solved.

 

[leesteken]

class    │ device │ id            │ iommugroup │ vendor │ device_name                                       
╞══════════╪════════╪═══════════════╪════════════╪════════╪══════════════════════════════════════════════════════
│ 0x010185 │ 0xa1bc │ 0000:00:16.2  │         -1 │ 0x8086 │ C620 Series Chipset Family IDE Redirection         
├──────────┼────────┼───────────────┼────────────┼────────┼──────────────────────────────────────────────────────
│ 0x010400 │ 0x201d │ 0000:17:05.5  │         -1 │ 0x8086 │ Volume Management Device NVMe RAID Controller     
├──────────┼────────┼───────────────┼────────────┼────────┼──────────────────────────────────────────────────────
│ 0x010601 │ 0xa182 │ 0000:00:17.0  │         -1 │ 0x8086 │ C620 Series Chipset Family SATA Controller [AHCI mode
├──────────┼────────┼───────────────┼────────────┼────────┼──────────────────────────────────────────────────────
│ 0x010802 │ 0x5006 │ 0000:5b:00.0  │         -1 │ 0x15b7 │ WD Black SN750 / PC SN730 NVMe SSD                 
├──────────┼────────┼───────────────┼────────────┼────────┼──────────────────────────────────────────────────────
│ 0x010802 │ 0x5016 │ 0000:5c:00.0  │         -1 │ 0x2646 │                                                   
├──────────┼────────┼───────────────┼────────────┼────────┼──────────────────────────────────────────────────────
│ 0x010802 │ 0x5016 │ 0000:5d:00.0  │         -1 │ 0x2646 │                                                   
├──────────┼────────┼───────────────┼────────────┼────────┼──────────────────────────────────────────────────────
│ 0x010802 │ 0x0116 │ 10000:01:00.0 │         -1 │ 0x1179 │ XG5 NVMe SSD Controller                           
├──────────┼────────┼───────────────┼────────────┼────────┼──────────────────────────────────────────────────────
│ 0x020000 │ 0x8125 │ 0000:02:00.0  │         -1 │ 0x10ec │ RTL8125 2.5GbE Controller                         
├──────────┼────────┼───────────────┼────────────┼────────┼──────────────────────────────────────────────────────
│ 0x030000 │ 0x2484 │ 0000:18:00.0  │         -1 │ 0x10de │ GA104 [GeForce RTX 3070]                           
├──────────┼────────┼───────────────┼────────────┼────────┼──────────────────────────────────────────────────────
│

The list is longer, but doesn't this indicate that I have IOMMU groups? And I can see my graphics card on this list.

No because all the group numbers (fourth column) are -1 (instead of 1, 2, 3, etc.).

I ended up switching the slot of my card and changing to UEFI boot instead of using GRUB. That fixed it an basically invalidates this whole thread. Marking as solved.

Great! I don't know why this would fix it but maybe it's just a DELL BIOS thing.