Unable to do PCI Passthrough with Dell R630 server?

victorhooi

Active Member
Apr 3, 2018
250
20
38
37
Hi,

I am trying to do PCI Passthrough per the wiki article:

https://pve.proxmox.com/wiki/Pci_passthrough

I am running Proxmox 6.0 on a Dell R630 server, with an Intel E5-2696 v4 CPU, and two X520-DA2 NICs.

I have added the following to /etc/defaults/grub:

Code:
GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on"
I then ran update-grub.

I also have the following in /etc/modules:
Code:
# /etc/modules: kernel modules to load at boot time.
#
# This file contains the names of kernel modules that should be loaded
# at boot time, one per line. Lines beginning with "#" are ignored.
vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd
I then rebooted the host.

This is from the dmesg output:
Code:
root@angusmtv-mon01:~# dmesg | grep -e DMAR -e IOMMU
[    0.013430] ACPI: DMAR 0x000000007BAFE000 0000D0 (v01 DELL   PE_SC3   00000001 DELL 00000001)
[    1.059589] DMAR: Host address width 46
[    1.059591] DMAR: DRHD base: 0x000000fbffc000 flags: 0x0
[    1.059602] DMAR: dmar0: reg_base_addr fbffc000 ver 1:0 cap 8d2078c106f0466 ecap f020df
[    1.059605] DMAR: DRHD base: 0x000000c7ffc000 flags: 0x1
[    1.059609] DMAR: dmar1: reg_base_addr c7ffc000 ver 1:0 cap 8d2078c106f0466 ecap f020df
[    1.059611] DMAR: ATSR flags: 0x0
[    1.059613] DMAR: ATSR flags: 0x0
[    1.059616] DMAR-IR: IOAPIC id 10 under DRHD base  0xfbffc000 IOMMU 0
[    1.059618] DMAR-IR: IOAPIC id 8 under DRHD base  0xc7ffc000 IOMMU 1
[    1.059620] DMAR-IR: IOAPIC id 9 under DRHD base  0xc7ffc000 IOMMU 1
[    1.059621] DMAR-IR: HPET id 0 under DRHD base 0xc7ffc000
[    1.059623] DMAR-IR: x2apic is disabled because BIOS sets x2apic opt out bit.
[    1.059624] DMAR-IR: Use 'intremap=no_x2apic_optout' to override the BIOS setting.
[    1.060310] DMAR-IR: Enabled IRQ remapping in xapic mode
However, the find command returns no IOMMU groups:
Code:
root@angusmtv-mon01:~# find /sys/kernel/iommu_groups/ -type l
Output of lspci:

See https://gist.github.com/victorhooi/bdef8f6157c3f6358ccf6469637da6f7 (forum complained it was too long

Any ideas what's going on, or what I need to do to get PCI passthrough working here?

Thanks,
Victor
 
Hi,
Did you enable the IOMMU in the Bios?
 
Is there normally a separate setting for IOMMU

There's a setting for VT-d (default to on). That is currently on.

And there's a setting for SR-IOV (default to off). I've enabled that.

I did see another setting for "X2APIC" mode that is currently disabled. Is that related at all to IOMMU?
 
I tried enabling the "X2APIC" setting, and rebooting.

Still nothing coming up under IOMMU groups - very strange.
Code:
root@angusmtv-mon01:~# dmesg | grep -e DMAR -e IOMMU
[    0.012962] ACPI: DMAR 0x000000007BAFE000 0000D0 (v01 DELL   PE_SC3   00000001 DELL 00000001)
[    1.058586] DMAR: Host address width 46
[    1.058588] DMAR: DRHD base: 0x000000fbffc000 flags: 0x0
[    1.058600] DMAR: dmar0: reg_base_addr fbffc000 ver 1:0 cap 8d2078c106f0466 ecap f020df
[    1.058602] DMAR: DRHD base: 0x000000c7ffc000 flags: 0x1
[    1.058606] DMAR: dmar1: reg_base_addr c7ffc000 ver 1:0 cap 8d2078c106f0466 ecap f020df
[    1.058609] DMAR: ATSR flags: 0x0
[    1.058610] DMAR: ATSR flags: 0x0
[    1.058613] DMAR-IR: IOAPIC id 10 under DRHD base  0xfbffc000 IOMMU 0
[    1.058615] DMAR-IR: IOAPIC id 8 under DRHD base  0xc7ffc000 IOMMU 1
[    1.058616] DMAR-IR: IOAPIC id 9 under DRHD base  0xc7ffc000 IOMMU 1
[    1.058618] DMAR-IR: HPET id 0 under DRHD base 0xc7ffc000
[    1.058620] DMAR-IR: Queued invalidation will be enabled to support x2apic and Intr-remapping.
[    1.058935] DMAR-IR: IRQ remapping was enabled on dmar0 but we are not in kdump mode
[    1.059211] DMAR-IR: IRQ remapping was enabled on dmar1 but we are not in kdump mode
[    1.059339] DMAR-IR: Enabled IRQ remapping in x2apic mode
root@angusmtv-mon01:~# find /sys/kernel/iommu_groups/ -type l
root@angusmtv-mon01:~#
What do you think might be going on?
 
I don't know the bios of the R630.
The problem is there are many names for this.
I would ask Dell what you have to enable in your Bios for IOMMU.
 
I'm, not 100% sure but a Dell R630 should support IOMMU.
The tricky thing is you must maybe enable 2 or more options in the Bios to get this work.
 
Old thread, but if anyone happens across this from a search I just spent half a day troubleshooting on a T430.
I have the 3 above options enabled in the BIOS, and then added the intel_iommu=on to grub. I realised later that the system isn't using grub due to ZFS / efi and added the command to the kernel command line /etc/kernel/cmdline as per the wiki.
When this still didn't work I eventually realised that the kernel commandline can only contain one line of commmands.

nano /etc/kernel/cmdline
Code:
root=ZFS=rpool/ROOT/pve-1 boot=zfs intel_iommu=on

T430 (and I assume also R630) supporting IOMMU.
 
  • Like
Reactions: McMarkis and rsiera
Old thread, but if anyone happens across this from a search I just spent half a day troubleshooting on a T430.
I have the 3 above options enabled in the BIOS, and then added the intel_iommu=on to grub. I realised later that the system isn't using grub due to ZFS / efi and added the command to the kernel command line /etc/kernel/cmdline as per the wiki.
When this still didn't work I eventually realised that the kernel commandline can only contain one line of commmands.

nano /etc/kernel/cmdline
Code:
root=ZFS=rpool/ROOT/pve-1 boot=zfs intel_iommu=on

T430 (and I assume also R630) supporting IOMMU.
This also worked on Dell R420, just remember to run proxmox-boot-tool refresh after modifying /etc/kernel/cmdline
 
It works here on R630 ( proxmox v8 )as well after adding the below cmd to /etc/kernel/cmdline

root=ZFS=rpool/ROOT/pve-1 boot=zfs intel_iommu=on
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!