Unable to do PCI Passthrough with Dell R630 server?

[victorhooi]

Hi,

I am trying to do PCI Passthrough per the wiki article:

https://pve.proxmox.com/wiki/Pci_passthrough

I am running Proxmox 6.0 on a Dell R630 server, with an Intel E5-2696 v4 CPU, and two X520-DA2 NICs.

I have added the following to /etc/defaults/grub:

GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on"

I then ran update-grub.

I also have the following in /etc/modules:

# /etc/modules: kernel modules to load at boot time.
#
# This file contains the names of kernel modules that should be loaded
# at boot time, one per line. Lines beginning with "#" are ignored.
vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd

I then rebooted the host.

This is from the dmesg output:

root@angusmtv-mon01:~# dmesg | grep -e DMAR -e IOMMU
[    0.013430] ACPI: DMAR 0x000000007BAFE000 0000D0 (v01 DELL   PE_SC3   00000001 DELL 00000001)
[    1.059589] DMAR: Host address width 46
[    1.059591] DMAR: DRHD base: 0x000000fbffc000 flags: 0x0
[    1.059602] DMAR: dmar0: reg_base_addr fbffc000 ver 1:0 cap 8d2078c106f0466 ecap f020df
[    1.059605] DMAR: DRHD base: 0x000000c7ffc000 flags: 0x1
[    1.059609] DMAR: dmar1: reg_base_addr c7ffc000 ver 1:0 cap 8d2078c106f0466 ecap f020df
[    1.059611] DMAR: ATSR flags: 0x0
[    1.059613] DMAR: ATSR flags: 0x0
[    1.059616] DMAR-IR: IOAPIC id 10 under DRHD base  0xfbffc000 IOMMU 0
[    1.059618] DMAR-IR: IOAPIC id 8 under DRHD base  0xc7ffc000 IOMMU 1
[    1.059620] DMAR-IR: IOAPIC id 9 under DRHD base  0xc7ffc000 IOMMU 1
[    1.059621] DMAR-IR: HPET id 0 under DRHD base 0xc7ffc000
[    1.059623] DMAR-IR: x2apic is disabled because BIOS sets x2apic opt out bit.
[    1.059624] DMAR-IR: Use 'intremap=no_x2apic_optout' to override the BIOS setting.
[    1.060310] DMAR-IR: Enabled IRQ remapping in xapic mode

However, the find command returns no IOMMU groups:

root@angusmtv-mon01:~# find /sys/kernel/iommu_groups/ -type l

Output of lspci:

See https://gist.github.com/victorhooi/bdef8f6157c3f6358ccf6469637da6f7 (forum complained it was too long

Any ideas what's going on, or what I need to do to get PCI passthrough working here?

Thanks,
Victor

 

[wolfgang]

Hi,
Did you enable the IOMMU in the Bios?

 

[victorhooi]

Is there normally a separate setting for IOMMU

There's a setting for VT-d (default to on). That is currently on.

And there's a setting for SR-IOV (default to off). I've enabled that.

I did see another setting for "X2APIC" mode that is currently disabled. Is that related at all to IOMMU?

 

[victorhooi]

I tried enabling the "X2APIC" setting, and rebooting.

Still nothing coming up under IOMMU groups - very strange.

root@angusmtv-mon01:~# dmesg | grep -e DMAR -e IOMMU
[    0.012962] ACPI: DMAR 0x000000007BAFE000 0000D0 (v01 DELL   PE_SC3   00000001 DELL 00000001)
[    1.058586] DMAR: Host address width 46
[    1.058588] DMAR: DRHD base: 0x000000fbffc000 flags: 0x0
[    1.058600] DMAR: dmar0: reg_base_addr fbffc000 ver 1:0 cap 8d2078c106f0466 ecap f020df
[    1.058602] DMAR: DRHD base: 0x000000c7ffc000 flags: 0x1
[    1.058606] DMAR: dmar1: reg_base_addr c7ffc000 ver 1:0 cap 8d2078c106f0466 ecap f020df
[    1.058609] DMAR: ATSR flags: 0x0
[    1.058610] DMAR: ATSR flags: 0x0
[    1.058613] DMAR-IR: IOAPIC id 10 under DRHD base  0xfbffc000 IOMMU 0
[    1.058615] DMAR-IR: IOAPIC id 8 under DRHD base  0xc7ffc000 IOMMU 1
[    1.058616] DMAR-IR: IOAPIC id 9 under DRHD base  0xc7ffc000 IOMMU 1
[    1.058618] DMAR-IR: HPET id 0 under DRHD base 0xc7ffc000
[    1.058620] DMAR-IR: Queued invalidation will be enabled to support x2apic and Intr-remapping.
[    1.058935] DMAR-IR: IRQ remapping was enabled on dmar0 but we are not in kdump mode
[    1.059211] DMAR-IR: IRQ remapping was enabled on dmar1 but we are not in kdump mode
[    1.059339] DMAR-IR: Enabled IRQ remapping in x2apic mode
root@angusmtv-mon01:~# find /sys/kernel/iommu_groups/ -type l
root@angusmtv-mon01:~#

What do you think might be going on?

 

[wolfgang]

I don't know the bios of the R630.
The problem is there are many names for this.
I would ask Dell what you have to enable in your Bios for IOMMU.

 

[victorhooi]

From reading online - I think

https://community.mellanox.com/s/ar...eredge-r730-bios-parameters-to-support-sr-iov

However, I'm not sure if the R630 supports IOMMU?

Anyhow - the use-case for this was to run ntopNG in a VM - I wanted to pass through a NIC, with one of those ports connected to a SPAN (port mirroring) port on our switch.

However, is there maybe another way of achieving this, with acceptable performance?

 

[wolfgang]

I'm, not 100% sure but a Dell R630 should support IOMMU.
The tricky thing is you must maybe enable 2 or more options in the Bios to get this work.

 

[i_am_jam]

Old thread, but if anyone happens across this from a search I just spent half a day troubleshooting on a T430.
I have the 3 above options enabled in the BIOS, and then added the intel_iommu=on to grub. I realised later that the system isn't using grub due to ZFS / efi and added the command to the kernel command line /etc/kernel/cmdline as per the wiki.
When this still didn't work I eventually realised that the kernel commandline can only contain one line of commmands.

nano /etc/kernel/cmdline

root=ZFS=rpool/ROOT/pve-1 boot=zfs intel_iommu=on

T430 (and I assume also R630) supporting IOMMU.

 

[McMarkis]

Old thread, but if anyone happens across this from a search I just spent half a day troubleshooting on a T430.
I have the 3 above options enabled in the BIOS, and then added the intel_iommu=on to grub. I realised later that the system isn't using grub due to ZFS / efi and added the command to the kernel command line /etc/kernel/cmdline as per the wiki.
When this still didn't work I eventually realised that the kernel commandline can only contain one line of commmands.

nano /etc/kernel/cmdline

root=ZFS=rpool/ROOT/pve-1 boot=zfs intel_iommu=on

T430 (and I assume also R630) supporting IOMMU.

This also worked on Dell R420, just remember to run proxmox-boot-tool refresh after modifying /etc/kernel/cmdline

 

[marvaen7]

It works here on R630 ( proxmox v8 )as well after adding the below cmd to /etc/kernel/cmdline

root=ZFS=rpool/ROOT/pve-1 boot=zfs intel_iommu=on

 

[andrewross3]

Does this work with the dell poweredge r430 as well?

 

[tunhube]

Just an update here, I had to rewrite from boot to root to make it work.