MinisForum NAB6 IOMMU direct passthrough problem

G

guanana

New Member
Jun 28, 2023
4
2
3
Hi,

I tried to to read as many pages and manuals as I could but I came down to the conclusion that my motherboard doesn't support IOMMU. If someone here could help me it will be great, but after 2 days of searching and trying I have to say I'm giving up a bit.

I bought my new MinisForum NAB6 16 x 12th Gen Intel(R) Core(TM) i7-12650H (1 Socket) and installed ProxMox on it. So far so good. All drivers are detected, all looks more than okay. I can spin up VMs an CTs.

I'm using it as a home server and I wanted to install Emby and of course I thought about GPU acceleration to give an extra boost to the decoding. The model I got doesn't have a super powerful GPU but it's better than nothing.

I tried to follow all the tutorials and guides to enable IOMMU. After a lot of pain and tests I found a command that checks for a lot of virtualisation capabitlities of your hardware `virt-host-validate` part of package `libvirt-clients`.

Output:
Bash: 
root@pve1:~# virt-host-validate
  QEMU: Checking for hardware virtualization                                 : PASS
  QEMU: Checking if device /dev/kvm exists                                   : PASS
  QEMU: Checking if device /dev/kvm is accessible                            : PASS
  QEMU: Checking if device /dev/vhost-net exists                             : PASS
  QEMU: Checking if device /dev/net/tun exists                               : PASS
  QEMU: Checking for cgroup 'cpu' controller support                         : PASS
  QEMU: Checking for cgroup 'cpuacct' controller support                     : PASS
  QEMU: Checking for cgroup 'cpuset' controller support                      : PASS
  QEMU: Checking for cgroup 'memory' controller support                      : PASS
  QEMU: Checking for cgroup 'devices' controller support                     : PASS
  QEMU: Checking for cgroup 'blkio' controller support                       : PASS
  QEMU: Checking for device assignment IOMMU support                         : WARN (No ACPI DMAR table found, IOMMU either disabled in BIOS or not supported by this hardware platform)
  QEMU: Checking for secure guest support                                    : WARN (Unknown if this platform has Secure Guest support)
   LXC: Checking for Linux >= 2.6.26                                         : PASS
   LXC: Checking for namespace ipc                                           : PASS
   LXC: Checking for namespace mnt                                           : PASS
   LXC: Checking for namespace pid                                           : PASS
   LXC: Checking for namespace uts                                           : PASS
   LXC: Checking for namespace net                                           : PASS
   LXC: Checking for namespace user                                          : PASS
   LXC: Checking for cgroup 'cpu' controller support                         : PASS
   LXC: Checking for cgroup 'cpuacct' controller support                     : PASS
   LXC: Checking for cgroup 'cpuset' controller support                      : PASS
   LXC: Checking for cgroup 'memory' controller support                      : PASS
   LXC: Checking for cgroup 'devices' controller support                     : PASS
   LXC: Checking for cgroup 'freezer' controller support                     : FAIL (Enable 'freezer' in kernel Kconfig file or mount/enable cgroup controller in your system)
   LXC: Checking for cgroup 'blkio' controller support                       : PASS
   LXC: Checking if device /sys/fs/fuse/connections exists                   : PASS

I tried many combinations
/etc/kernel/cmdline

boot=zfs intel_iommu=on iommu=pt nox2apic intremap=no_x2apic_optout video=efifb:off,vesafb:off
Bash: 
pve1 kernel: DMAR: IOMMU enabled
pve1 kernel: iommu: Default domain type: Passthrough (set via kernel command line)
pve1 kernel: i915 0000:00:02.0: vgaarb: deactivate vga console
pve1 kernel: i915 0000:00:02.0: [drm] Using Transparent Hugepages
pve1 kernel: i915 0000:00:02.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=io+mem:owns=io+mem
pve1 kernel: i915 0000:00:02.0: [drm] Finished loading DMC firmware i915/adlp_dmc_ver2_16.bin (v2.16)
pve1 kernel: i915 0000:00:02.0: [drm] GuC firmware i915/adlp_guc_70.bin version 70.5.1
pve1 kernel: i915 0000:00:02.0: [drm] HuC firmware i915/tgl_huc.bin version 7.9.3
pve1 kernel: i915 0000:00:02.0: [drm] HuC authenticated
pve1 kernel: i915 0000:00:02.0: [drm] GuC submission enabled
pve1 kernel: i915 0000:00:02.0: [drm] GuC SLPC enabled
pve1 kernel: i915 0000:00:02.0: [drm] GuC RC: enabled
pve1 kernel: i915 0000:00:02.0: [drm] Protected Xe Path (PXP) protected content support initialized
pve1 kernel: [drm] Initialized i915 1.6.0 20201103 for 0000:00:02.0 on minor 0
pve1 kernel: ACPI: video: Video Device [GFX0] (multi-head: yes  rom: no  post: no)
pve1 kernel: i915 0000:00:02.0: [drm] Cannot find any crtc or sizes
boot=zfs intel_iommu=on i915.enable_guc=3 i915.max_vfs=7
Bash: 
pve1 kernel: Reserving Intel graphics memory at [mem 0x4c800000-0x503fffff]
pve1 kernel: DMAR: IOMMU enabled
pve1 kernel: pci 0000:00:02.0: DMAR: Skip IOMMU disabling for graphics
pve1 kernel: i915 0000:00:02.0: vgaarb: deactivate vga console
pve1 kernel: i915 0000:00:02.0: [drm] Using Transparent Hugepages
pve1 kernel: i915 0000:00:02.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=io+mem:owns=io+mem
pve1 kernel: i915 0000:00:02.0: [drm] Finished loading DMC firmware i915/adlp_dmc_ver2_16.bin (v2.16)
pve1 kernel: i915 0000:00:02.0: [drm] GuC firmware i915/adlp_guc_70.bin version 70.5.1
pve1 kernel: i915 0000:00:02.0: [drm] HuC firmware i915/tgl_huc.bin version 7.9.3
pve1 kernel: i915 0000:00:02.0: [drm] HuC authenticated
pve1 kernel: i915 0000:00:02.0: [drm] GuC submission enabled
pve1 kernel: i915 0000:00:02.0: [drm] GuC SLPC enabled
pve1 kernel: i915 0000:00:02.0: [drm] GuC RC: enabled
pve1 kernel: i915 0000:00:02.0: [drm] Protected Xe Path (PXP) protected content support initialized
pve1 kernel: [drm] Initialized i915 1.6.0 20201103 for 0000:00:02.0 on minor 0
pve1 kernel: ACPI: video: Video Device [GFX0] (multi-head: yes  rom: no  post: no)
pve1 kernel: i915 0000:00:02.0: driver does not support SR-IOV configuration via sysfs
boot=zfs intel_iommu=on iommu=pt i915.max_vfs=7
Bash: 
pve1 kernel: DMAR: IOMMU enabled
pve1 kernel: ACPI: Enabled 7 GPEs in block 00 to 7F
pve1 kernel: pci 0000:00:02.0: DMAR: Skip IOMMU disabling for graphics

pve1 kernel: pci 0000:00:02.0: vgaarb: setting as boot VGA device
pve1 kernel: pci 0000:00:02.0: vgaarb: bridge control possible
pve1 kernel: pci 0000:00:02.0: vgaarb: VGA device added: decodes=io+mem,owns=io+mem,locks=none
pve1 kernel: pnp: PnP ACPI: found 7 devices
pve1 kernel: Disabling lock debugging due to kernel taint
pve1 kernel: vfio-pci 0000:00:02.0: vgaarb: deactivate vga console
pve1 kernel: vfio-pci 0000:00:02.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=io+mem:owns=io+mem
pve1 kernel: vfio-pci: probe of 0000:00:02.0 failed with error -22
pve1 kernel: i915 0000:00:02.0: Running in SR-IOV PF mode
pve1 kernel: i915 0000:00:02.0: vgaarb: deactivate vga console
pve1 kernel: i915 0000:00:02.0: [drm] Using Transparent Hugepages
pve1 kernel: i915 0000:00:02.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=io+mem:owns=io+mem
pve1 kernel: i915 0000:00:02.0: Direct firmware load for i915/adlp_dmc.bin failed with error -2
pve1 kernel: i915 0000:00:02.0: [drm] Finished loading DMC firmware i915/adlp_dmc_ver2_16.bin (v2.16)
pve1 kernel: intel_rapl_msr: PL4 support detected.
pve1 kernel: intel_rapl_common: Found RAPL domain package
pve1 kernel: intel_rapl_common: Found RAPL domain core
pve1 kernel: intel_rapl_common: Found RAPL domain uncore
pve1 kernel: i915 0000:00:02.0: [drm] GT0: GuC firmware i915/adlp_guc_70.bin version 70.5.1
pve1 kernel: i915 0000:00:02.0: [drm] GT0: HuC firmware i915/tgl_huc.bin version 7.9.3
pve1 kernel: i915 0000:00:02.0: [drm] GT0: HuC: authenticated for all workloads!
pve1 kernel: i915 0000:00:02.0: [drm] GT0: GUC: submission enabled
pve1 kernel: i915 0000:00:02.0: [drm] GT0: GUC: SLPC enabled
pve1 kernel: i915 0000:00:02.0: [drm] GuC RC: enabled
pve1 kernel: i915 0000:00:02.0: [drm] Protected Xe Path (PXP) protected content support initialized
pve1 kernel: [drm] Initialized i915 1.6.0 20201103 for 0000:00:02.0 on minor 0
pve1 kernel: ACPI: video: Video Device [GFX0] (multi-head: yes  rom: no  post: no)

So after some of these tests I came to the conclusion that SR-IOV worked but not IOMMU. I don't know if it makes much sense but it seems to be the case for this computer. I ended up being able to see the 7 devices under the GPU 02.00.[1-7] but still I was unable to assign it to a VM due to IOMMU not being detected and it failing every time I tried (even when I set it to RAW device or PCI-E, etc).

I tried to filter a bit the output to make the post more readable. If anybody has any idea of how to do more tests I'm all ears, if not at least I hope this post saves valuable hours to the next guy that buys MinisForum NAB6 and finds that passtrhought doesn't work because of IOMMU.
 
guanana said:
/etc/kernel/cmdline
boot=zfs intel_iommu=on iommu=pt nox2apic intremap=no_x2apic_optout video=efifb:off,vesafb:off
Click to expand...
I did not read you post in detail, but video=efifb:off,vesafb:off has not workedf or some time (and is invalid as you should write it as video=efifb:off video=vesafb:off but that also had not worked for Proxmox since it switched to simplefb). If you want to passthrough the boot/single GPU then you need this work-around.
Sorry, but I have no experience with your hardware or your software, so I don't expect to be much help beyond the boot/single GPU work-around.
 
I guess, you try to virtualize your intel gpu, iris xe in your case.

What i don't understand is, that
Code: 
i915.max_vfs=7
Spins up 7 virtual GPU's at all.

While the driver clearly says:
Code: 
driver does not support SR-IOV configuration via sysfs

To shrink your hope further, as far i know, Intel GPUs doesn't support SR-IOV at all.
And yes, i remember myself that intel wrote somewhere that everything above 11th or 12th gen supports SR-IOV.
But they don't!

I think that they wanted to make a successor to GVT-g/GVT-d, but i didn't seen any development or statements about that.

What you can do is passing your igpu to one VM or passing one igpu to multiple CT's.

About iommu, it's usually enough to activate in your bios vt-d (that's basically iommu)
And adding iommu=pt to kernel cmdline.
intel_iommu=on is already default since kernel 6.2 (i think), or at least with kernel 6.2 for sure if your hardware supports it.
Just iommu=pt is needed in some cases and vt-d enabled bios.

And the vfio modules in /etc/modules
And if you want to passthrough your igpu to a VM, you need to blacklist the i915 driver and bind vfio-pci to the device onboot + rebuilding initramfs.

Cheers
 
Thanks for the replies. I think I made SR-IOV work. If I understand it properly that technology is precisly the one that allowed me to be able to see several GPUs (split 1 physical into 7 virtual). That theoretically gives better performance and allows for easier sharing. Unfortunately here I'm hiting another problem. I cannot activate IOMMU. I tried with iommu-pt and other combinations but the hardware I got doesn't show any option in BIOS related to VT. Reading some posts from MinisForum they say that you don't need to enable anything because everything is enabled and HIDDEN by default. I cannot see anything in my BIOS. Pretty much I can configure boot order and not much more. It's true that VT-x seems enabled but VT-d seems like it's not.

Bash: 
[    0.100315] Kernel command line: initrd=\EFI\proxmox\6.2.16-3-pve\initrd.img-6.2.16-3-pve root=ZFS=rpool/ROOT/pve-1 boot=zfs intel_iommu=on iommu=pt i915.enable_guc=3 i915.max_vfs=7
[    0.100373] DMAR: IOMMU enabled
[    0.606048] pci 0000:00:02.0: DMAR: Skip IOMMU disabling for graphics
[    0.649126] iommu: Default domain type: Passthrough (set via kernel command line)
[    3.919918] pci 0000:00:02.1: DMAR: Skip IOMMU disabling for graphics
[    3.920056] pci 0000:00:02.2: DMAR: Skip IOMMU disabling for graphics
[    3.920106] pci 0000:00:02.3: DMAR: Skip IOMMU disabling for graphics
[    3.920154] pci 0000:00:02.4: DMAR: Skip IOMMU disabling for graphics
[    3.920201] pci 0000:00:02.5: DMAR: Skip IOMMU disabling for graphics
[    3.920255] pci 0000:00:02.6: DMAR: Skip IOMMU disabling for graphics
[    3.920301] pci 0000:00:02.7: DMAR: Skip IOMMU disabling for graphics
[    3.049948] i915 0000:00:02.0: Running in SR-IOV PF mode
[    3.050427] i915 0000:00:02.0: vgaarb: deactivate vga console
[    3.050467] i915 0000:00:02.0: [drm] Using Transparent Hugepages
[    3.050832] i915 0000:00:02.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=io+mem:owns=io+mem
[    3.051253] i915 0000:00:02.0: Direct firmware load for i915/adlp_dmc.bin failed with error -2
[    3.053289] i915 0000:00:02.0: [drm] Finished loading DMC firmware i915/adlp_dmc_ver2_16.bin (v2.16)
[    3.058794] i915 0000:00:02.0: [drm] GT0: GuC firmware i915/adlp_guc_70.bin version 70.5.1
[    3.058799] i915 0000:00:02.0: [drm] GT0: HuC firmware i915/tgl_huc.bin version 7.9.3
[    3.074101] i915 0000:00:02.0: [drm] GT0: HuC: authenticated for all workloads!
[    3.074584] i915 0000:00:02.0: [drm] GT0: GUC: submission enabled
[    3.074586] i915 0000:00:02.0: [drm] GT0: GUC: SLPC enabled
[    3.074992] i915 0000:00:02.0: [drm] GuC RC: enabled
[    3.075494] i915 0000:00:02.0: [drm] Protected Xe Path (PXP) protected content support initialized
[    3.076741] [drm] Initialized i915 1.6.0 20201103 for 0000:00:02.0 on minor 0
[    3.077961] snd_hda_intel 0000:00:1f.3: bound 0000:00:02.0 (ops i915_audio_component_bind_ops [i915])
[    3.078086] i915 0000:00:02.0: 7 VFs could be associated with this PF
[    3.080466] i915 0000:00:02.0: [drm] Cannot find any crtc or sizes
[    3.080660] i915 0000:00:02.0: [drm] Cannot find any crtc or sizes
[    3.080761] i915 0000:00:02.0: [drm] Cannot find any crtc or sizes
[    3.919971] i915 0000:00:02.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=none:owns=io+mem
[    3.920085] i915 0000:00:02.0: vgaarb: changed VGA decodes: olddecodes=none,decodes=none:owns=io+mem
[    3.920133] i915 0000:00:02.0: vgaarb: changed VGA decodes: olddecodes=none,decodes=none:owns=io+mem
[    3.920181] i915 0000:00:02.0: vgaarb: changed VGA decodes: olddecodes=none,decodes=none:owns=io+mem
[    3.920234] i915 0000:00:02.0: vgaarb: changed VGA decodes: olddecodes=none,decodes=none:owns=io+mem
[    3.920281] i915 0000:00:02.0: vgaarb: changed VGA decodes: olddecodes=none,decodes=none:owns=io+mem
[    3.920330] i915 0000:00:02.0: vgaarb: changed VGA decodes: olddecodes=none,decodes=none:owns=io+mem
[    3.920368] i915 0000:00:02.0: Enabled 7 VFs

I can see the GPUs but IOMMU still showing as "off"
Screenshot 2023-06-29 at 00.12.48.png

If I remove the options i915.enable_guc=3 i915.max_vfs=7 and keep iommu=pt, etc... I still get same result (but without the multiple GPUs, just 1 device).
 
Bildschirmfoto 2023-06-29 um 02.13.36.pngBildschirmfoto 2023-06-29 um 02.14.59.png

Thats how its looking here.
One is an Intel NUC 13 (i3)
The Other is an X570d4i-2t + 5800x

For me it clearly looks like VT-d is definitively disabled on your board.

Cheers
 
Ramalama said:
View attachment 52330View attachment 52331

Thats how its looking here.
One is an Intel NUC 13 (i3)
The Other is an X570d4i-2t + 5800x

For me it clearly looks like VT-d is definitively disabled on your board.

Cheers
Click to expand...
Unfortunately it seems that way.... VT-x enabled by default in NAB6 but VT-d seems like it's not. Stupid decision since it seems that all the hardware is ready to support it.... but since there's no option to change in the BIOS I guess it's not much I can do. I'm thinking to return the NAB6 and get a new NUC. A bit more expensive but I'm not sure if all the headache is worth the price :(
 
For anybody facing the same issue... I can confirm it wasn't enabled in the BIOS. After a painful journey I got a BIOS update, enabled the option and now with the simple config:

Code: 
boot=zfs intel_iommu=on iommu=pt

I got it working

Code: 
[    0.102933] DMAR: IOMMU enabled
[    0.243790] DMAR: Host address width 39
[    0.243792] DMAR: DRHD base: 0x000000fed90000 flags: 0x0
[    0.243799] DMAR: dmar0: reg_base_addr fed90000 ver 4:0 cap 1c0000c40660462 ecap 29a00f0505e
[    0.243804] DMAR: DRHD base: 0x000000fed91000 flags: 0x1
[    0.243811] DMAR: dmar1: reg_base_addr fed91000 ver 5:0 cap d2008c40660462 ecap f050da
[    0.243815] DMAR: RMRR base: 0x0000004c000000 end: 0x000000503fffff
[    0.243819] DMAR-IR: IOAPIC id 2 under DRHD base  0xfed91000 IOMMU 1
[    0.243823] DMAR-IR: HPET id 0 under DRHD base 0xfed91000
[    0.243825] DMAR-IR: Queued invalidation will be enabled to support x2apic and Intr-remapping.
[    0.245705] DMAR-IR: Enabled IRQ remapping in x2apic mode
[    1.329759] pci 0000:00:02.0: DMAR: Skip IOMMU disabling for graphics
[    1.470999] iommu: Default domain type: Passthrough (set via kernel command line)
[    1.584718] DMAR: No ATSR found
[    1.584720] DMAR: No SATC found
[    1.584721] DMAR: IOMMU feature fl1gp_support inconsistent
[    1.584723] DMAR: IOMMU feature pgsel_inv inconsistent
[    1.584724] DMAR: IOMMU feature nwfs inconsistent
[    1.584726] DMAR: IOMMU feature dit inconsistent
[    1.584728] DMAR: IOMMU feature sc_support inconsistent
[    1.584730] DMAR: IOMMU feature dev_iotlb_support inconsistent
[    1.584732] DMAR: dmar0: Using Queued invalidation
[    1.584737] DMAR: dmar1: Using Queued invalidation
[    1.584919] pci 0000:00:02.0: Adding to iommu group 0
[    1.584950] pci 0000:00:00.0: Adding to iommu group 1
[    1.584963] pci 0000:00:14.0: Adding to iommu group 2
[    1.584970] pci 0000:00:14.2: Adding to iommu group 2
[    1.584980] pci 0000:00:16.0: Adding to iommu group 3
[    1.584988] pci 0000:00:17.0: Adding to iommu group 4
[    1.585000] pci 0000:00:1c.0: Adding to iommu group 5
[    1.585012] pci 0000:00:1d.0: Adding to iommu group 6
[    1.585029] pci 0000:00:1d.1: Adding to iommu group 7
[    1.585044] pci 0000:00:1f.0: Adding to iommu group 8
[    1.585052] pci 0000:00:1f.3: Adding to iommu group 8
[    1.585059] pci 0000:00:1f.4: Adding to iommu group 8
[    1.585067] pci 0000:00:1f.5: Adding to iommu group 8
[    1.585081] pci 0000:01:00.0: Adding to iommu group 9
[    1.585094] pci 0000:02:00.0: Adding to iommu group 10
[    1.585106] pci 0000:03:00.0: Adding to iommu group 11
[    1.585206] DMAR: Intel(R) Virtualization Technology for Directed I/O

Not sure if this is worrying or not

Code: 
[    1.329759] pci 0000:00:02.0: DMAR: Skip IOMMU disabling for graphics

but I don't think so

If anybody is interested on how I solved it (the solution is VERY specific for MinisForum NAB6, you can check this post
https://www.reddit.com/r/MiniPCs/comments/14lnkh0/comment/jq5y8kv/?context=3
 
  • Like
Reactions: loomes and leesteken
DMAR: Skip IOMMU disabling for graphics

You can ignore that.
Congratulations
 
Hello :),
i also get a MinisForum NAB6 for Proxmox 8.0.

@guanana Thanks for your Informations.
My NAB6 was already on BIOS version 1.00D, which is why I was able to activate VT-d directly.
I do following:

1. I try to enable IOMMU (https://pve.proxmox.com/wiki/PCI_Passthrough#Verify_IOMMU_isolation)
But "find /sys/kernel/iommu_groups/ -type l" did not give any result

2. I add with "nano /etc/kernel/cmdline"
boot=zfs intel_iommu=on iommu=pt

//Im used ext4 at Proxmox installation, have i change boot=zfs?)
run update-initramfs-u -k all

3. I add with "nano /etc/modules" (I found this in another Proxmox manual)
Code: 
 vfio
 vfio_iommu_type1
 vfio_pci
 vfio_virqfd
run proxmox-boot-toolrefresh

4. After points 2 and 3 didn't help, I found the solution in a forum:
Edit "nano /etc/default/grub"
Code: 
GRUB_DEFAULT=0
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian`
#GRUB_CMDLINE_LINUX_DEFAULT="quiet"
GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on"                       # << I add this
GRUB_CMDLINE_LINUX=""
run update-grub
run proxmox-boot-tool refresh
run update-initramfs -u -k all

Now IOMMU works fine but the interface "enp3s0" doesn't work after i start my pfSense-VM .
When i use ifup enp3s0 i get the message: "warning: enp3s0: interface not recognized - please check interface configuration"
When i use ip a there is no interface enp3s0
At the Dashboard enp3s0 is inactive

When i reboot Proxmox, enp3s0 works until the start of pfSense-VM.
When i only start my Nexcloud-LXC and my Windows11-VM there is no problem with enp3s0

If I undo point 4, the interface enp3s0 works, but iommu doesn't work again.

The other Interface enp2s0 has no Problems.

Does anyone have any ideas? Did I miss something in the IOMMU configuration?
 

Attachments

  • 1688470491850.png
    1688470491850.png
    128 KB · Views: 25
Last edited:
Thanks for the tip. An old PCI device from the previous server was still attached to the VM. Seemingly randomly with the ID of enp3s0. Now that I've removed it, everything works. Thank you very much!
1688479328819.png
 
guanana said:
Thanks for the replies. I think I made SR-IOV work. If I understand it properly that technology is precisly the one that allowed me to be able to see several GPUs (split 1 physical into 7 virtual). That theoretically gives better performance and allows for easier sharing. Unfortunately here I'm hiting another problem. I cannot activate IOMMU. I tried with iommu-pt and other combinations but the hardware I got doesn't show any option in BIOS related to VT. Reading some posts from MinisForum they say that you don't need to enable anything because everything is enabled and HIDDEN by default. I cannot see anything in my BIOS. Pretty much I can configure boot order and not much more. It's true that VT-x seems enabled but VT-d seems like it's not.

Bash: 
[    0.100315] Kernel command line: initrd=\EFI\proxmox\6.2.16-3-pve\initrd.img-6.2.16-3-pve root=ZFS=rpool/ROOT/pve-1 boot=zfs intel_iommu=on iommu=pt i915.enable_guc=3 i915.max_vfs=7
[    0.100373] DMAR: IOMMU enabled
[    0.606048] pci 0000:00:02.0: DMAR: Skip IOMMU disabling for graphics
[    0.649126] iommu: Default domain type: Passthrough (set via kernel command line)
[    3.919918] pci 0000:00:02.1: DMAR: Skip IOMMU disabling for graphics
[    3.920056] pci 0000:00:02.2: DMAR: Skip IOMMU disabling for graphics
[    3.920106] pci 0000:00:02.3: DMAR: Skip IOMMU disabling for graphics
[    3.920154] pci 0000:00:02.4: DMAR: Skip IOMMU disabling for graphics
[    3.920201] pci 0000:00:02.5: DMAR: Skip IOMMU disabling for graphics
[    3.920255] pci 0000:00:02.6: DMAR: Skip IOMMU disabling for graphics
[    3.920301] pci 0000:00:02.7: DMAR: Skip IOMMU disabling for graphics
[    3.049948] i915 0000:00:02.0: Running in SR-IOV PF mode
[    3.050427] i915 0000:00:02.0: vgaarb: deactivate vga console
[    3.050467] i915 0000:00:02.0: [drm] Using Transparent Hugepages
[    3.050832] i915 0000:00:02.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=io+mem:owns=io+mem
[    3.051253] i915 0000:00:02.0: Direct firmware load for i915/adlp_dmc.bin failed with error -2
[    3.053289] i915 0000:00:02.0: [drm] Finished loading DMC firmware i915/adlp_dmc_ver2_16.bin (v2.16)
[    3.058794] i915 0000:00:02.0: [drm] GT0: GuC firmware i915/adlp_guc_70.bin version 70.5.1
[    3.058799] i915 0000:00:02.0: [drm] GT0: HuC firmware i915/tgl_huc.bin version 7.9.3
[    3.074101] i915 0000:00:02.0: [drm] GT0: HuC: authenticated for all workloads!
[    3.074584] i915 0000:00:02.0: [drm] GT0: GUC: submission enabled
[    3.074586] i915 0000:00:02.0: [drm] GT0: GUC: SLPC enabled
[    3.074992] i915 0000:00:02.0: [drm] GuC RC: enabled
[    3.075494] i915 0000:00:02.0: [drm] Protected Xe Path (PXP) protected content support initialized
[    3.076741] [drm] Initialized i915 1.6.0 20201103 for 0000:00:02.0 on minor 0
[    3.077961] snd_hda_intel 0000:00:1f.3: bound 0000:00:02.0 (ops i915_audio_component_bind_ops [i915])
[    3.078086] i915 0000:00:02.0: 7 VFs could be associated with this PF
[    3.080466] i915 0000:00:02.0: [drm] Cannot find any crtc or sizes
[    3.080660] i915 0000:00:02.0: [drm] Cannot find any crtc or sizes
[    3.080761] i915 0000:00:02.0: [drm] Cannot find any crtc or sizes
[    3.919971] i915 0000:00:02.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=none:owns=io+mem
[    3.920085] i915 0000:00:02.0: vgaarb: changed VGA decodes: olddecodes=none,decodes=none:owns=io+mem
[    3.920133] i915 0000:00:02.0: vgaarb: changed VGA decodes: olddecodes=none,decodes=none:owns=io+mem
[    3.920181] i915 0000:00:02.0: vgaarb: changed VGA decodes: olddecodes=none,decodes=none:owns=io+mem
[    3.920234] i915 0000:00:02.0: vgaarb: changed VGA decodes: olddecodes=none,decodes=none:owns=io+mem
[    3.920281] i915 0000:00:02.0: vgaarb: changed VGA decodes: olddecodes=none,decodes=none:owns=io+mem
[    3.920330] i915 0000:00:02.0: vgaarb: changed VGA decodes: olddecodes=none,decodes=none:owns=io+mem
[    3.920368] i915 0000:00:02.0: Enabled 7 VFs

I can see the GPUs but IOMMU still showing as "off"
View attachment 52329

If I remove the options i915.enable_guc=3 i915.max_vfs=7 and keep iommu=pt, etc... I still get same result (but without the multiple GPUs, just 1 device).
Click to expand...
Hello, can you explain how you made SR-IOV work?
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back