Unable to do PCI Passthrough with Dell R630 server?

V

victorhooi

Well-Known Member
Apr 3, 2018
255
20
58
38
Hi,

I am trying to do PCI Passthrough per the wiki article:

https://pve.proxmox.com/wiki/Pci_passthrough

I am running Proxmox 6.0 on a Dell R630 server, with an Intel E5-2696 v4 CPU, and two X520-DA2 NICs.

I have added the following to /etc/defaults/grub:

Code: 
GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on"
I then ran update-grub.

I also have the following in /etc/modules:
Code: 
# /etc/modules: kernel modules to load at boot time.
#
# This file contains the names of kernel modules that should be loaded
# at boot time, one per line. Lines beginning with "#" are ignored.
vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd
I then rebooted the host.

This is from the dmesg output:
Code: 
root@angusmtv-mon01:~# dmesg | grep -e DMAR -e IOMMU
[    0.013430] ACPI: DMAR 0x000000007BAFE000 0000D0 (v01 DELL   PE_SC3   00000001 DELL 00000001)
[    1.059589] DMAR: Host address width 46
[    1.059591] DMAR: DRHD base: 0x000000fbffc000 flags: 0x0
[    1.059602] DMAR: dmar0: reg_base_addr fbffc000 ver 1:0 cap 8d2078c106f0466 ecap f020df
[    1.059605] DMAR: DRHD base: 0x000000c7ffc000 flags: 0x1
[    1.059609] DMAR: dmar1: reg_base_addr c7ffc000 ver 1:0 cap 8d2078c106f0466 ecap f020df
[    1.059611] DMAR: ATSR flags: 0x0
[    1.059613] DMAR: ATSR flags: 0x0
[    1.059616] DMAR-IR: IOAPIC id 10 under DRHD base  0xfbffc000 IOMMU 0
[    1.059618] DMAR-IR: IOAPIC id 8 under DRHD base  0xc7ffc000 IOMMU 1
[    1.059620] DMAR-IR: IOAPIC id 9 under DRHD base  0xc7ffc000 IOMMU 1
[    1.059621] DMAR-IR: HPET id 0 under DRHD base 0xc7ffc000
[    1.059623] DMAR-IR: x2apic is disabled because BIOS sets x2apic opt out bit.
[    1.059624] DMAR-IR: Use 'intremap=no_x2apic_optout' to override the BIOS setting.
[    1.060310] DMAR-IR: Enabled IRQ remapping in xapic mode
However, the find command returns no IOMMU groups:
Code: 
root@angusmtv-mon01:~# find /sys/kernel/iommu_groups/ -type l
Output of lspci:

See https://gist.github.com/victorhooi/bdef8f6157c3f6358ccf6469637da6f7 (forum complained it was too long

Any ideas what's going on, or what I need to do to get PCI passthrough working here?

Thanks,
Victor
 
Hi,
Did you enable the IOMMU in the Bios?
 
Is there normally a separate setting for IOMMU

There's a setting for VT-d (default to on). That is currently on.

And there's a setting for SR-IOV (default to off). I've enabled that.

I did see another setting for "X2APIC" mode that is currently disabled. Is that related at all to IOMMU?
 
I tried enabling the "X2APIC" setting, and rebooting.

Still nothing coming up under IOMMU groups - very strange.
Code: 
root@angusmtv-mon01:~# dmesg | grep -e DMAR -e IOMMU
[    0.012962] ACPI: DMAR 0x000000007BAFE000 0000D0 (v01 DELL   PE_SC3   00000001 DELL 00000001)
[    1.058586] DMAR: Host address width 46
[    1.058588] DMAR: DRHD base: 0x000000fbffc000 flags: 0x0
[    1.058600] DMAR: dmar0: reg_base_addr fbffc000 ver 1:0 cap 8d2078c106f0466 ecap f020df
[    1.058602] DMAR: DRHD base: 0x000000c7ffc000 flags: 0x1
[    1.058606] DMAR: dmar1: reg_base_addr c7ffc000 ver 1:0 cap 8d2078c106f0466 ecap f020df
[    1.058609] DMAR: ATSR flags: 0x0
[    1.058610] DMAR: ATSR flags: 0x0
[    1.058613] DMAR-IR: IOAPIC id 10 under DRHD base  0xfbffc000 IOMMU 0
[    1.058615] DMAR-IR: IOAPIC id 8 under DRHD base  0xc7ffc000 IOMMU 1
[    1.058616] DMAR-IR: IOAPIC id 9 under DRHD base  0xc7ffc000 IOMMU 1
[    1.058618] DMAR-IR: HPET id 0 under DRHD base 0xc7ffc000
[    1.058620] DMAR-IR: Queued invalidation will be enabled to support x2apic and Intr-remapping.
[    1.058935] DMAR-IR: IRQ remapping was enabled on dmar0 but we are not in kdump mode
[    1.059211] DMAR-IR: IRQ remapping was enabled on dmar1 but we are not in kdump mode
[    1.059339] DMAR-IR: Enabled IRQ remapping in x2apic mode
root@angusmtv-mon01:~# find /sys/kernel/iommu_groups/ -type l
root@angusmtv-mon01:~#
What do you think might be going on?
 
I don't know the bios of the R630.
The problem is there are many names for this.
I would ask Dell what you have to enable in your Bios for IOMMU.
 
I'm, not 100% sure but a Dell R630 should support IOMMU.
The tricky thing is you must maybe enable 2 or more options in the Bios to get this work.
 
Old thread, but if anyone happens across this from a search I just spent half a day troubleshooting on a T430.
I have the 3 above options enabled in the BIOS, and then added the intel_iommu=on to grub. I realised later that the system isn't using grub due to ZFS / efi and added the command to the kernel command line /etc/kernel/cmdline as per the wiki.
When this still didn't work I eventually realised that the kernel commandline can only contain one line of commmands.

nano /etc/kernel/cmdline
Code: 
root=ZFS=rpool/ROOT/pve-1 boot=zfs intel_iommu=on

T430 (and I assume also R630) supporting IOMMU.
 
  • Like
Reactions: tunhube, McMarkis and rsiera
i_am_jam said:
Old thread, but if anyone happens across this from a search I just spent half a day troubleshooting on a T430.
I have the 3 above options enabled in the BIOS, and then added the intel_iommu=on to grub. I realised later that the system isn't using grub due to ZFS / efi and added the command to the kernel command line /etc/kernel/cmdline as per the wiki.
When this still didn't work I eventually realised that the kernel commandline can only contain one line of commmands.

nano /etc/kernel/cmdline
Code: 
root=ZFS=rpool/ROOT/pve-1 boot=zfs intel_iommu=on

T430 (and I assume also R630) supporting IOMMU.
Click to expand...
This also worked on Dell R420, just remember to run proxmox-boot-tool refresh after modifying /etc/kernel/cmdline
 
It works here on R630 ( proxmox v8 )as well after adding the below cmd to /etc/kernel/cmdline

root=ZFS=rpool/ROOT/pve-1 boot=zfs intel_iommu=on
 
You must log in or register to reply here.
Top Bottom