PCI(e) Passthrough is not working

U

user332255

New Member
Sep 5, 2021
24
5
3
29
Hi,
I want to passthrough a NVME to one of my VMs. Therefore, I have to enable IOMMU. I tried to follow the guide, but unfortunately it isn't working.
Enabling VT-d in BIOS -> check
/etc/modules =
Code: 
vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd
update-initramfs -u -k all
/proc/cmdline =
Code: 
initrd=\EFI\proxmox\5.15.64-1-pve\initrd.img-5.15.64-1-pve root=ZFS=rpool/ROOT/pve-1 boot=zfs
/etc/kernel/cmdline =
Code: 
root=ZFS=rpool/ROOT/pve-1 boot=zfs quiet intel_iommu=on
pve-efiboot-tool refresh
proxmox-boot-tool refresh
reboot
dmesg | grep -e DMAR -e IOMMU -e AMD-Vi =
Code: 
[    0.008204] ACPI: DMAR 0x0000000089ED8CC0 0000C8 (v01 INTEL  EDK2     00000002      01000013)
[    0.008230] ACPI: Reserving DMAR table memory at [mem 0x89ed8cc0-0x89ed8d87]
[    0.190147] DMAR: Host address width 39
[    0.190149] DMAR: DRHD base: 0x000000fed90000 flags: 0x0
[    0.190154] DMAR: dmar0: reg_base_addr fed90000 ver 1:0 cap 1c0000c40660462 ecap 19e2ff0505e
[    0.190157] DMAR: DRHD base: 0x000000fed91000 flags: 0x1
[    0.190161] DMAR: dmar1: reg_base_addr fed91000 ver 1:0 cap d2008c40660462 ecap f050da
[    0.190164] DMAR: RMRR base: 0x0000003e2e0000 end: 0x0000003e2fffff
[    0.190166] DMAR: [Firmware Bug]: No firmware reserved region can cover this RMRR [0x000000003e2e0000-0x000000003e2fffff], contact BIOS vendor for fixes
[    0.190169] DMAR: [Firmware Bug]: Your BIOS is broken; bad RMRR [0x000000003e2e0000-0x000000003e2fffff]
[    0.190173] DMAR: RMRR base: 0x0000008b800000 end: 0x0000008fffffff
[    0.190175] DMAR: RMRR base: 0x00000089dee000 end: 0x00000089e6dfff
[    0.190177] DMAR-IR: IOAPIC id 2 under DRHD base  0xfed91000 IOMMU 1
[    0.190179] DMAR-IR: HPET id 0 under DRHD base 0xfed91000
[    0.190181] DMAR-IR: Queued invalidation will be enabled to support x2apic and Intr-remapping.
[    0.193336] DMAR-IR: Enabled IRQ remapping in x2apic mode
As you can see, dmesg | grep -e DMAR -e IOMMU -e AMD-Vi should display that IOMMU, Directed I/O or Interrupt Remapping is enabled (depending on hardware and kernel).
I also tried for the grub version, but the problem remains the same. Can you help me solve this problem?

My server's hardware:
CPU: Intel i3-8100
Mainboard: Asus WS C246 PRO
RAM: 48 DDR4 ECC UDIMM
 
  • Like
Reactions: Jez2022
user332255 said:
[ 0.190166] DMAR: [Firmware Bug]: No firmware reserved region can cover this RMRR [0x000000003e2e0000-0x000000003e2fffff], contact BIOS vendor for fixes [ 0.190169] DMAR: [Firmware Bug]: Your BIOS is broken; bad RMRR [0x000000003e2e0000-0x000000003e2fffff]
Click to expand...
There are some threads on this forum about RMRR problems. I assume that Asus will tell you that they don't support Linux but maybe try a BIOS update?
 
leesteken said:
but maybe try a BIOS update?
Click to expand...
The BIOS update resolved this problem, but pci-e passthrough still isn't working:
Code: 
dmesg | grep -e DMAR -e IOMMU -e AMD-Vi
[    0.008057] ACPI: DMAR 0x000000008A2E94F8 0000C8 (v01 INTEL  EDK2     00000002      01000013)
[    0.008083] ACPI: Reserving DMAR table memory at [mem 0x8a2e94f8-0x8a2e95bf]
[    0.194287] DMAR: Host address width 39
[    0.194289] DMAR: DRHD base: 0x000000fed90000 flags: 0x0
[    0.194294] DMAR: dmar0: reg_base_addr fed90000 ver 1:0 cap 1c0000c40660462 ecap 19e2ff0505e
[    0.194297] DMAR: DRHD base: 0x000000fed91000 flags: 0x1
[    0.194300] DMAR: dmar1: reg_base_addr fed91000 ver 1:0 cap d2008c40660462 ecap f050da
[    0.194303] DMAR: RMRR base: 0x0000008a733000 end: 0x0000008a97cfff
[    0.194306] DMAR: RMRR base: 0x0000008b800000 end: 0x0000008fffffff
[    0.194307] DMAR: RMRR base: 0x0000008a1fc000 end: 0x0000008a27bfff
[    0.194310] DMAR-IR: IOAPIC id 2 under DRHD base  0xfed91000 IOMMU 1
[    0.194312] DMAR-IR: HPET id 0 under DRHD base 0xfed91000
[    0.194314] DMAR-IR: Queued invalidation will be enabled to support x2apic and Intr-remapping.
[    0.195904] DMAR-IR: Enabled IRQ remapping in x2apic mode
 
user332255 said:
The BIOS update resolved this problem, but pci-e passthrough still isn't working:
Code: 
dmesg | grep -e DMAR -e IOMMU -e AMD-Vi
[    0.008057] ACPI: DMAR 0x000000008A2E94F8 0000C8 (v01 INTEL  EDK2     00000002      01000013)
[    0.008083] ACPI: Reserving DMAR table memory at [mem 0x8a2e94f8-0x8a2e95bf]
[    0.194287] DMAR: Host address width 39
[    0.194289] DMAR: DRHD base: 0x000000fed90000 flags: 0x0
[    0.194294] DMAR: dmar0: reg_base_addr fed90000 ver 1:0 cap 1c0000c40660462 ecap 19e2ff0505e
[    0.194297] DMAR: DRHD base: 0x000000fed91000 flags: 0x1
[    0.194300] DMAR: dmar1: reg_base_addr fed91000 ver 1:0 cap d2008c40660462 ecap f050da
[    0.194303] DMAR: RMRR base: 0x0000008a733000 end: 0x0000008a97cfff
[    0.194306] DMAR: RMRR base: 0x0000008b800000 end: 0x0000008fffffff
[    0.194307] DMAR: RMRR base: 0x0000008a1fc000 end: 0x0000008a27bfff
[    0.194310] DMAR-IR: IOAPIC id 2 under DRHD base  0xfed91000 IOMMU 1
[    0.194312] DMAR-IR: HPET id 0 under DRHD base 0xfed91000
[    0.194314] DMAR-IR: Queued invalidation will be enabled to support x2apic and Intr-remapping.
[    0.195904] DMAR-IR: Enabled IRQ remapping in x2apic mode
Click to expand...
I don't know how to tell from this. Do you have multiple IOMMU groups (which is the definite test for IOMMU enablement)? for d in /sys/kernel/iommu_groups/*/devices/*; do n=${d#*/iommu_groups/*}; n=${n%%/*}; printf 'IOMMU group %s ' "$n"; lspci -nns "${d##*/}"; done
 
leesteken said:
for d in /sys/kernel/iommu_groups/*/devices/*; do n=${d#*/iommu_groups/*}; n=${n%%/*}; printf 'IOMMU group %s ' "$n"; lspci -nns "${d##*/}"; done
Click to expand...
Code: 
IOMMU group * 00:00.0 Host bridge [0600]: Intel Corporation 8th Gen Core 4-core Desktop Processor Host Bridge/DRAM Registers [Coffee Lake S] [8086:3e1f] (rev 08)
00:01.0 PCI bridge [0604]: Intel Corporation 6th-10th Gen Core Processor PCIe Controller (x16) [8086:1901] (rev 08)
00:02.0 VGA compatible controller [0300]: Intel Corporation CoffeeLake-S GT2 [UHD Graphics 630] [8086:3e91]
00:14.0 USB controller [0c03]: Intel Corporation Cannon Lake PCH USB 3.1 xHCI Host Controller [8086:a36d] (rev 10)
00:14.2 RAM memory [0500]: Intel Corporation Cannon Lake PCH Shared SRAM [8086:a36f] (rev 10)
00:16.0 Communication controller [0780]: Intel Corporation Cannon Lake PCH HECI Controller [8086:a360] (rev 10)
00:17.0 SATA controller [0106]: Intel Corporation Cannon Lake PCH SATA AHCI Controller [8086:a352] (rev 10)
00:1d.0 PCI bridge [0604]: Intel Corporation Cannon Lake PCH PCI Express Root Port #11 [8086:a332] (rev f0)
00:1f.0 ISA bridge [0601]: Intel Corporation Cannon Point-LP LPC Controller [8086:a309] (rev 10)
00:1f.3 Audio device [0403]: Intel Corporation Cannon Lake PCH cAVS [8086:a348] (rev 10)
00:1f.4 SMBus [0c05]: Intel Corporation Cannon Lake PCH SMBus Controller [8086:a323] (rev 10)
00:1f.5 Serial bus controller [0c80]: Intel Corporation Cannon Lake PCH SPI Controller [8086:a324] (rev 10)
00:1f.6 Ethernet controller [0200]: Intel Corporation Ethernet Connection (7) I219-LM [8086:15bb] (rev 10)
01:00.0 Serial Attached SCSI controller [0107]: Broadcom / LSI SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon] [1000:0072] (rev 03)
02:00.0 Ethernet controller [0200]: Intel Corporation I210 Gigabit Network Connection [8086:1533] (rev 03)

This is the result
 
user332255 said:
Code: 
IOMMU group * 00:00.0 Host bridge [0600]: Intel Corporation 8th Gen Core 4-core Desktop Processor Host Bridge/DRAM Registers [Coffee Lake S] [8086:3e1f] (rev 08)
00:01.0 PCI bridge [0604]: Intel Corporation 6th-10th Gen Core Processor PCIe Controller (x16) [8086:1901] (rev 08)
00:02.0 VGA compatible controller [0300]: Intel Corporation CoffeeLake-S GT2 [UHD Graphics 630] [8086:3e91]
00:14.0 USB controller [0c03]: Intel Corporation Cannon Lake PCH USB 3.1 xHCI Host Controller [8086:a36d] (rev 10)
00:14.2 RAM memory [0500]: Intel Corporation Cannon Lake PCH Shared SRAM [8086:a36f] (rev 10)
00:16.0 Communication controller [0780]: Intel Corporation Cannon Lake PCH HECI Controller [8086:a360] (rev 10)
00:17.0 SATA controller [0106]: Intel Corporation Cannon Lake PCH SATA AHCI Controller [8086:a352] (rev 10)
00:1d.0 PCI bridge [0604]: Intel Corporation Cannon Lake PCH PCI Express Root Port #11 [8086:a332] (rev f0)
00:1f.0 ISA bridge [0601]: Intel Corporation Cannon Point-LP LPC Controller [8086:a309] (rev 10)
00:1f.3 Audio device [0403]: Intel Corporation Cannon Lake PCH cAVS [8086:a348] (rev 10)
00:1f.4 SMBus [0c05]: Intel Corporation Cannon Lake PCH SMBus Controller [8086:a323] (rev 10)
00:1f.5 Serial bus controller [0c80]: Intel Corporation Cannon Lake PCH SPI Controller [8086:a324] (rev 10)
00:1f.6 Ethernet controller [0200]: Intel Corporation Ethernet Connection (7) I219-LM [8086:15bb] (rev 10)
01:00.0 Serial Attached SCSI controller [0107]: Broadcom / LSI SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon] [1000:0072] (rev 03)
02:00.0 Ethernet controller [0200]: Intel Corporation I210 Gigabit Network Connection [8086:1533] (rev 03)

This is the result
Click to expand...
There is only one group (*), so IOMMU is not enabled. Double check the BIOS settings (VT-d for Intel) and cat /proc/cmdline.
 
leesteken said:
Double check the BIOS settings (VT-d for Intel)
Click to expand...
I did:
Code: 
Intel (VMX) Virtualization Technology = Enabled
VT-d = Enabled
Above 4G Decoding = Enabled/ Disabled (Tested both)
leesteken said:
cat /proc/cmdline
Click to expand...
Code: 
initrd=\EFI\proxmox\5.15.64-1-pve\initrd.img-5.15.64-1-pve root=ZFS=rpool/ROOT/pve-1 boot=zfs

Are there any other settings in BIOS, which could solve this problem?
 
user332255 said:
Code: 
initrd=\EFI\proxmox\5.15.64-1-pve\initrd.img-5.15.64-1-pve root=ZFS=rpool/ROOT/pve-1 boot=zfs
Click to expand...
The intel_iommu=on is missing, which prevents IOMMU from enabling. Your change to /etc/kernel/cmdline was not applied. Usually running proxmox-boot-tool refresh and rebooting should fix this.
 
leesteken said:
/etc/kernel/cmdline
Click to expand...
Code: 
root=ZFS=rpool/ROOT/pve-1 boot=zfs quiet intel_iommu=on
leesteken said:
proxmox-boot-tool refresh
Click to expand...
Code: 
Running hook script 'proxmox-auto-removal'..
Running hook script 'zz-proxmox-boot'..
Re-executing '/etc/kernel/postinst.d/zz-proxmox-boot' in new private mount namespace..
No root= parameter in /etc/kernel/cmdline found!

Or should I apply the changes directly via /proc/cmdline?
 
user332255 said:
Or should I apply the changes directly via /proc/cmdline?
Click to expand...
No, and that is not possible. I'm quite sure you need to edit /etc/kernel/cmdline.
user332255 said:
Code: 
Running hook script 'proxmox-auto-removal'..
Running hook script 'zz-proxmox-boot'..
Re-executing '/etc/kernel/postinst.d/zz-proxmox-boot' in new private mount namespace..
No root= parameter in /etc/kernel/cmdline found!
Click to expand...
This is strange. Maybe you added an empty line to the beginning of the file? Everything needs to be on the first (and only) line in the /etc/kernel/cmdline file. Can you please attach the actual file to your (next) post?
 
leesteken said:
Everything needs to be on the first (and only) line in the /etc/kernel/cmdline
Click to expand...
THANK YOU! This solved the problem. I had some lines commented out before the first line (I kept the original line as a backup). I copied the command root=ZFS=rpool/ROOT/pve-1 boot=zfs quiet intel_iommu=on in the first line and moved the comments after that. Now I get following responses:
cat /proc/cmdline
Code: 
initrd=\EFI\proxmox\5.15.64-1-pve\initrd.img-5.15.64-1-pve root=ZFS=rpool/ROOT/pve-1 boot=zfs quiet intel_iommu=on

dmesg | grep -e DMAR -e IOMMU -e AMD-Vi
Code: 
[    0.007683] ACPI: DMAR 0x000000008A2E94F8 0000C8 (v01 INTEL  EDK2     00000002      01000013)
[    0.007705] ACPI: Reserving DMAR table memory at [mem 0x8a2e94f8-0x8a2e95bf]
[    0.072192] DMAR: IOMMU enabled
[    0.193772] DMAR: Host address width 39
[    0.193773] DMAR: DRHD base: 0x000000fed90000 flags: 0x0
[    0.193777] DMAR: dmar0: reg_base_addr fed90000 ver 1:0 cap 1c0000c40660462 ecap 19e2ff0505e
[    0.193779] DMAR: DRHD base: 0x000000fed91000 flags: 0x1
[    0.193781] DMAR: dmar1: reg_base_addr fed91000 ver 1:0 cap d2008c40660462 ecap f050da
[    0.193783] DMAR: RMRR base: 0x0000008a733000 end: 0x0000008a97cfff
[    0.193784] DMAR: RMRR base: 0x0000008b800000 end: 0x0000008fffffff
[    0.193785] DMAR: RMRR base: 0x0000008a1fc000 end: 0x0000008a27bfff
[    0.193787] DMAR-IR: IOAPIC id 2 under DRHD base  0xfed91000 IOMMU 1
[    0.193788] DMAR-IR: HPET id 0 under DRHD base 0xfed91000
[    0.193789] DMAR-IR: Queued invalidation will be enabled to support x2apic and Intr-remapping.
[    0.195376] DMAR-IR: Enabled IRQ remapping in x2apic mode
[    0.511107] DMAR: No ATSR found
[    0.511108] DMAR: No SATC found
[    0.511109] DMAR: IOMMU feature fl1gp_support inconsistent
[    0.511110] DMAR: IOMMU feature pgsel_inv inconsistent
[    0.511111] DMAR: IOMMU feature nwfs inconsistent
[    0.511111] DMAR: IOMMU feature pasid inconsistent
[    0.511112] DMAR: IOMMU feature eafs inconsistent
[    0.511113] DMAR: IOMMU feature prs inconsistent
[    0.511113] DMAR: IOMMU feature nest inconsistent
[    0.511114] DMAR: IOMMU feature mts inconsistent
[    0.511114] DMAR: IOMMU feature sc_support inconsistent
[    0.511115] DMAR: IOMMU feature dev_iotlb_support inconsistent
[    0.511116] DMAR: dmar0: Using Queued invalidation
[    0.511118] DMAR: dmar1: Using Queued invalidation
[    0.512596] DMAR: Intel(R) Virtualization Technology for Directed I/O

for d in /sys/kernel/iommu_groups/*/devices/*; do n=${d#*/iommu_groups/*}; n=${n%%/*}; printf 'IOMMU group %s ' "$n"; lspci -nns "${d##*/}"; done
Code: 
IOMMU group 0 00:00.0 Host bridge [0600]: Intel Corporation 8th Gen Core 4-core Desktop Processor Host Bridge/DRAM Registers [Coffee Lake S] [8086:3e1f] (rev 08)
IOMMU group 1 00:01.0 PCI bridge [0604]: Intel Corporation 6th-10th Gen Core Processor PCIe Controller (x16) [8086:1901] (rev 08)
IOMMU group 1 01:00.0 Serial Attached SCSI controller [0107]: Broadcom / LSI SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon] [1000:0072] (rev 03)
IOMMU group 2 00:02.0 VGA compatible controller [0300]: Intel Corporation CoffeeLake-S GT2 [UHD Graphics 630] [8086:3e91]
IOMMU group 3 00:14.0 USB controller [0c03]: Intel Corporation Cannon Lake PCH USB 3.1 xHCI Host Controller [8086:a36d] (rev 10)
IOMMU group 3 00:14.2 RAM memory [0500]: Intel Corporation Cannon Lake PCH Shared SRAM [8086:a36f] (rev 10)
IOMMU group 4 00:16.0 Communication controller [0780]: Intel Corporation Cannon Lake PCH HECI Controller [8086:a360] (rev 10)
IOMMU group 5 00:17.0 SATA controller [0106]: Intel Corporation Cannon Lake PCH SATA AHCI Controller [8086:a352] (rev 10)
IOMMU group 6 00:1d.0 PCI bridge [0604]: Intel Corporation Cannon Lake PCH PCI Express Root Port #11 [8086:a332] (rev f0)
IOMMU group 7 00:1f.0 ISA bridge [0601]: Intel Corporation Cannon Point-LP LPC Controller [8086:a309] (rev 10)
IOMMU group 7 00:1f.3 Audio device [0403]: Intel Corporation Cannon Lake PCH cAVS [8086:a348] (rev 10)
IOMMU group 7 00:1f.4 SMBus [0c05]: Intel Corporation Cannon Lake PCH SMBus Controller [8086:a323] (rev 10)
IOMMU group 7 00:1f.5 Serial bus controller [0c80]: Intel Corporation Cannon Lake PCH SPI Controller [8086:a324] (rev 10)
IOMMU group 7 00:1f.6 Ethernet controller [0200]: Intel Corporation Ethernet Connection (7) I219-LM [8086:15bb] (rev 10)
IOMMU group 8 02:00.0 Ethernet controller [0200]: Intel Corporation I210 Gigabit Network Connection [8086:1533] (rev 03)

However, if I try to passthrough my HBA LSI card to the VM, I get the error message:
Code: 
start failed: QEMU exited with code 1

EDIT:
What can I do with this error code?
Click to expand...
I accidently had one of the drive already passed through via sata, so i removed it and the error message went away.
Now, I'm struggling with a timeout during the boot of the VM, but I think PCI passthrough is working now!
 
Last edited:
If you don't see errors in the Proxmox GUI Syslog (or use journalctl) from around the time that you tried starting the VM, then it usually is a memory issue.
With passthrough all memory must be pinned into actual host RAM because of PCI(e) DMA. Try reducing the amount of memory for the VM.
Attach the VM configuration file to your post, if you want me to have a look.
 
leesteken said:
Try reducing the amount of memory for the VM.
Click to expand...
I assigned 8GB (enabled ballooning) to my VM. My Server has 48GB RAM.

I'm running TrueNAS Scale in the VM and therefore I want to passthrough my drives to the vm.
During boot, I see that my LSI HBA card is recognized, with all attached Sata drives to it. Unfortunately, TrueNAS tries to boot from them, and I get an endless loop, because no bootable device was found. I already disabled the HBA card for booting in proxmox, nevertheless TrueNAS tries to boot from them.

Attach the VM configuration file to your post, if you want me to have a look.
Click to expand...
Can you tell me the command please? Thank you
 
Last edited:
user332255 said:
'm running TrueNAS Scale in the VM and therefore I want to passthrough my drives to the vm.
During boot, I see that my LSI HBA card is recognized, with all attached Sata drives to it. Unfortunately, TrueNAS tries to boot from them, and I get an endless loop, because no bootable device was found. I already disabled the HBA card for booting in proxmox, nevertheless TrueNAS tries to boot from them.
Click to expand...
Okay, passthrough works. I have no idea how to configure TrueNAS to boot from the right drive.
user332255 said:
Can you tell me the command please? Thank you
Click to expand...
The VM configurations files can be found in the /etc/pve/qemu-server/ directory. But no need as it looks like the problem is not in the VM configuration.
 
user332255 said:
I get an endless loop, because no bootable device was found
Click to expand...
I found the solution: https://forum.proxmox.com/threads/w..._-with-lsi-hba-passthrough.111736/post-482285
From what I understand, ROM-BAR is the PCIe card's ROM "Base Address Register." When this is selected, the firmware ROM on the PCIe card will run during the VM's boot process. When it is not enabled (not selected), then the board's firmware will not run during the VM POST. I've had to disable this to get HBAs to passthrough and work in the VM.
Click to expand...
So I disabled ROM-BAR and now it seems to work! I'll keep you updated, if something will not work, again.

Thanks alot for your help, @leesteken !
 
Last edited:
  • Like
Reactions: leesteken
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back