PCIe Passthrough

[Mineralwasser]

Hi,

I've just configured my first Proxmox installation.
Now I want to setup my NAS VM.
For that I need to passthrough a PCIe HBA card to the VM.

I've followed the steps listed in this wiki article.

Here's what I did:

1. Verified that VT-d is enabled in BIOS.
2. Added the following options to /etc/default/grub
   

   GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt"

3. update-grub
4. Added the following to /etc/modules
   

   fio
   vfio_iommu_type1
   vfio_pci
   vfio_virqfd

5. update-initramfs -u -k all
6. Reboot
7. dmesg | grep -e DMAR -e IOMMU
   

   [    0.008441] ACPI: DMAR 0x000000008C53CC38 000070 (v01 INTEL  EDK2     00000002      01000013)
   [    0.273244] DMAR: Host address width 39
   [    0.273245] DMAR: DRHD base: 0x000000fed91000 flags: 0x1
   [    0.273250] DMAR: dmar0: reg_base_addr fed91000 ver 1:0 cap d2008c40660462 ecap f050da
   [    0.273252] DMAR: RMRR base: 0x0000008ceb1000 end: 0x0000008d0fafff
   [    0.273254] DMAR-IR: IOAPIC id 2 under DRHD base  0xfed91000 IOMMU 0
   [    0.273255] DMAR-IR: HPET id 0 under DRHD base 0xfed91000
   [    0.273257] DMAR-IR: Queued invalidation will be enabled to support x2apic and Intr-remapping.
   [    0.276203] DMAR-IR: Enabled IRQ remapping in x2apic mode

In the Wiki the next step is Verify IOMMU Isolation.

When I run find /sys/kernel/iommu_groups/ -type l there isn't any output.
Also ls /sys/kernel/iommu_groups/ doesn't return anything.

If I try to passthrough the card anyway, I get "Error: IOMMU not present" if I try to start the VM.

Hope anyone can help me.

Mainboard: Supermicro X11SCH-LN4F
CPU: Xeon E-2146G

 

[Mineralwasser]

Oh well... my bad.

I need to add the options to /etc/kernel/cmdline and run pve-efiboot-tool refresh instead of editing the GRUB options.

I guess I'll just leave the options in both places.

Now dmesg | grep -e DMAR -e IOMMU returns the additional line [ 0.154822] DMAR: IOMMU enabled.

And find /sys/kernel/iommu_groups/ -type l returns useful stuff.

 

[Mineralwasser]

Okay, now there is the next problem...

As stated in the documentation, I created the following file:

# /etc/modprobe.d/passthrough.conf
options vfio-pci ids=1000:0097

The numbers I aquired through lspci -nn which returns the following for my device:

01:00.0 Serial Attached SCSI controller [0107]: LSI Logic / Symbios Logic SAS3008 PCI-Express Fusion-MPT SAS-3 [1000:0097] (rev 02)

Then I did update-initramfs -u -k all and rebooted the machine.

lspci -nnk should now either return "Kernel driver in use: vfio-pci" or nothing. But that's not the case here. It still returns the same driver as before:

01:00.0 Serial Attached SCSI controller [0107]: LSI Logic / Symbios Logic SAS3008 PCI-Express Fusion-MPT SAS-3 [1000:0097] (rev 02)
        Subsystem: LSI Logic / Symbios Logic SAS9300-8i [1000:30e0]
        Kernel driver in use: mpt3sas
        Kernel modules: mpt3sas

There is also mentioned another method which is blacklisting the driver completely.

For this I should also create a .conf file in /etc/modprobe.d/.

In my case the content would be blacklist mpt3sas right?
But even if I do that, after a reboot it is still using this driver.... (???)

 

[peter100]

Also struggling with PCI passthru (although my initial forum post was titled "USB Sound card passthru to Windows 10"). However, I think I need to passthru a USB Controller instead.

When the system is started, I get:

05:00.0 USB controller [0c03]: Renesas Technology Corp. uPD720202 USB 3.0 Host Controller [1912:0015] (rev 02)
Subsystem: Renesas Technology Corp. uPD720202 USB 3.0 Host Controller [1912:0015]
Kernel driver in use: xhci_hcd

But, when I'm starting the guest (Windows, in my case) it changes to:

05:00.0 USB controller [0c03]: Renesas Technology Corp. uPD720202 USB 3.0 Host Controller [1912:0015] (rev 02)
Subsystem: Renesas Technology Corp. uPD720202 USB 3.0 Host Controller [1912:0015]
Kernel driver in use: vfio-pci

Hope this can help you

Peter

 

[Mineralwasser]

I'll try. Thank you.

Another little question regarding passthrough:
When I try to add the PCIe device through the Proxmox GUI there is a little checkbox called "All Functions".
I'm not sure what this is? There is no explanation in the Proxmox documentation.

edit:

I did both:

# /etc/modprobe.d/passthrough.conf
blacklist mpt3sas
options vfio-pci ids=1000:0097

It works now.

 

[j4ys0n]

I'm currently using a proxmox node as NAS, but I set up ZFS RAID, with cache, in proxmox and then mounted that array in a turnkey file server container to share on the network. works really well. Probably a lot less hassle than doing PCIe passthrough. You get the full capacity of the underlying ZFS pool.

 

[vdarko]

Thank you for this post. It helped solve my problem of setting up TrueNAS Core 12.0 on ProxMox.

In the end, had to remove a checkmark from ROM-Bar under advanced for PCIe HBA to pass through successfully.

This is how I did it >> Enable PCIe Passthrough ProxMox 6.2-12

# add the options "quiet intel_iommu=on iommu=pt" to > /etc/kernel/cmdline
# line should look something like this > root=ZFS=rpool/ROOT/pve-1 boot=zfs quiet intel_iommu=on iommu=pt

# and run > pve-efiboot-tool refresh

# add to the > /etc/modules

fio
vfio_pci
vfio_virqfd

# and run > update-initramfs -u -k all

# run > lspci -nn to identify device ID's, in my case "1000:0097":

01:00.0 Serial Attached SCSI controller [0107]: LSI Logic / Symbios Logic SAS3008 PCI-Express Fusion-MPT SAS-3 [1000:0097] (rev 02)
02:00.0 Serial Attached SCSI controller [0107]: LSI Logic / Symbios Logic SAS3008 PCI-Express Fusion-MPT SAS-3 [1000:0097] (rev 02)

# create/edit file, add:
nano /etc/modprobe.d/passthrough.conf

blacklist mpt3sas
options vfio-pci ids=1000:0097

# run > lspci -nnk to identify device Kernel modules, in my case "mpt3sas":

01:00.0 Serial Attached SCSI controller [0107]: LSI Logic / Symbios Logic SAS3008 PCI-Express Fusion-MPT SAS-3 [1000:0097] (rev 02)
        Subsystem: Super Micro Computer Inc SAS3008 PCI-Express Fusion-MPT SAS-3 (AOC-S3008L-L8e) [15d9:0808]
        Kernel driver in use: vfio-pci
        Kernel modules: mpt3sas
02:00.0 Serial Attached SCSI controller [0107]: LSI Logic / Symbios Logic SAS3008 PCI-Express Fusion-MPT SAS-3 [1000:0097] (rev 02)
        Subsystem: Dell HBA330 Adapter [1028:1f45]
        Kernel driver in use: vfio-pci
        Kernel modules: mpt3sas

# create/edit file, add:
nano /etc/modprobe.d/pve-blacklist.conf

blacklist nvidiafb (# was there already)
blacklist mpt3sas

# create/edit file, add:
# or use command > echo "options vfio_iommu_type1 allow_unsafe_interrupts=1" > /etc/modprobe.d/iommu_unsafe_interrupts.conf
nano /etc/modprobe.d/iommu_unsafe_interrupts.conf

options vfio_iommu_type1 allow_unsafe_interrupts=1

# and run > update-initramfs -u -k all

Reboot

# also, edited file to look like this:
nano /etc/default/grub.d/init-select.cfg

GRUB_CMDLINE_LINUX_DEFAULT="${GRUB_CMDLINE_LINUX_DEFAULT} intel_iommu=on"

run > dmesg | grep -e DMAR -e IOMMU (# returns > ... 0.541356] DMAR: IOMMU enabled)
run > find /sys/kernel/iommu_groups/ -type l (# returns > /sys/kernel/iommu_groups/17/devices/0000:00:04.0 ...)

Not sure if all steps are needed but it works.