Unable to do PCI Passthrough with Dell R630 server?

victorhooi

Member
Apr 3, 2018
174
11
23
33
Hi,

I am trying to do PCI Passthrough per the wiki article:

https://pve.proxmox.com/wiki/Pci_passthrough

I am running Proxmox 6.0 on a Dell R630 server, with an Intel E5-2696 v4 CPU, and two X520-DA2 NICs.

I have added the following to /etc/defaults/grub:

Code:
GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on"
I then ran update-grub.

I also have the following in /etc/modules:
Code:
# /etc/modules: kernel modules to load at boot time.
#
# This file contains the names of kernel modules that should be loaded
# at boot time, one per line. Lines beginning with "#" are ignored.
vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd
I then rebooted the host.

This is from the dmesg output:
Code:
root@angusmtv-mon01:~# dmesg | grep -e DMAR -e IOMMU
[    0.013430] ACPI: DMAR 0x000000007BAFE000 0000D0 (v01 DELL   PE_SC3   00000001 DELL 00000001)
[    1.059589] DMAR: Host address width 46
[    1.059591] DMAR: DRHD base: 0x000000fbffc000 flags: 0x0
[    1.059602] DMAR: dmar0: reg_base_addr fbffc000 ver 1:0 cap 8d2078c106f0466 ecap f020df
[    1.059605] DMAR: DRHD base: 0x000000c7ffc000 flags: 0x1
[    1.059609] DMAR: dmar1: reg_base_addr c7ffc000 ver 1:0 cap 8d2078c106f0466 ecap f020df
[    1.059611] DMAR: ATSR flags: 0x0
[    1.059613] DMAR: ATSR flags: 0x0
[    1.059616] DMAR-IR: IOAPIC id 10 under DRHD base  0xfbffc000 IOMMU 0
[    1.059618] DMAR-IR: IOAPIC id 8 under DRHD base  0xc7ffc000 IOMMU 1
[    1.059620] DMAR-IR: IOAPIC id 9 under DRHD base  0xc7ffc000 IOMMU 1
[    1.059621] DMAR-IR: HPET id 0 under DRHD base 0xc7ffc000
[    1.059623] DMAR-IR: x2apic is disabled because BIOS sets x2apic opt out bit.
[    1.059624] DMAR-IR: Use 'intremap=no_x2apic_optout' to override the BIOS setting.
[    1.060310] DMAR-IR: Enabled IRQ remapping in xapic mode
However, the find command returns no IOMMU groups:
Code:
root@angusmtv-mon01:~# find /sys/kernel/iommu_groups/ -type l
Output of lspci:

See https://gist.github.com/victorhooi/bdef8f6157c3f6358ccf6469637da6f7 (forum complained it was too long

Any ideas what's going on, or what I need to do to get PCI passthrough working here?

Thanks,
Victor
 

wolfgang

Proxmox Staff Member
Staff member
Oct 1, 2014
5,619
377
103
Hi,
Did you enable the IOMMU in the Bios?
 

victorhooi

Member
Apr 3, 2018
174
11
23
33
Is there normally a separate setting for IOMMU

There's a setting for VT-d (default to on). That is currently on.

And there's a setting for SR-IOV (default to off). I've enabled that.

I did see another setting for "X2APIC" mode that is currently disabled. Is that related at all to IOMMU?
 

victorhooi

Member
Apr 3, 2018
174
11
23
33
I tried enabling the "X2APIC" setting, and rebooting.

Still nothing coming up under IOMMU groups - very strange.
Code:
root@angusmtv-mon01:~# dmesg | grep -e DMAR -e IOMMU
[    0.012962] ACPI: DMAR 0x000000007BAFE000 0000D0 (v01 DELL   PE_SC3   00000001 DELL 00000001)
[    1.058586] DMAR: Host address width 46
[    1.058588] DMAR: DRHD base: 0x000000fbffc000 flags: 0x0
[    1.058600] DMAR: dmar0: reg_base_addr fbffc000 ver 1:0 cap 8d2078c106f0466 ecap f020df
[    1.058602] DMAR: DRHD base: 0x000000c7ffc000 flags: 0x1
[    1.058606] DMAR: dmar1: reg_base_addr c7ffc000 ver 1:0 cap 8d2078c106f0466 ecap f020df
[    1.058609] DMAR: ATSR flags: 0x0
[    1.058610] DMAR: ATSR flags: 0x0
[    1.058613] DMAR-IR: IOAPIC id 10 under DRHD base  0xfbffc000 IOMMU 0
[    1.058615] DMAR-IR: IOAPIC id 8 under DRHD base  0xc7ffc000 IOMMU 1
[    1.058616] DMAR-IR: IOAPIC id 9 under DRHD base  0xc7ffc000 IOMMU 1
[    1.058618] DMAR-IR: HPET id 0 under DRHD base 0xc7ffc000
[    1.058620] DMAR-IR: Queued invalidation will be enabled to support x2apic and Intr-remapping.
[    1.058935] DMAR-IR: IRQ remapping was enabled on dmar0 but we are not in kdump mode
[    1.059211] DMAR-IR: IRQ remapping was enabled on dmar1 but we are not in kdump mode
[    1.059339] DMAR-IR: Enabled IRQ remapping in x2apic mode
root@angusmtv-mon01:~# find /sys/kernel/iommu_groups/ -type l
root@angusmtv-mon01:~#
What do you think might be going on?
 

wolfgang

Proxmox Staff Member
Staff member
Oct 1, 2014
5,619
377
103
I don't know the bios of the R630.
The problem is there are many names for this.
I would ask Dell what you have to enable in your Bios for IOMMU.
 

wolfgang

Proxmox Staff Member
Staff member
Oct 1, 2014
5,619
377
103
I'm, not 100% sure but a Dell R630 should support IOMMU.
The tricky thing is you must maybe enable 2 or more options in the Bios to get this work.
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE and Proxmox Mail Gateway. We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get your own in 60 seconds.

Buy now!