Passthrough NVME fails, but there are no IOMMU conflits

H

HyggeJJ

New Member
Apr 22, 2025
4
0
1
Hi All,

I am trying to passthrough an Intel DC P4600 NVME PCIe drive, as a Raw Device but am so far unsuccesful.
Proxmox crashes as soon as I start the VM.
The same VM has a Broadcom 9500-8i passed through, which works perfectly.

I have tried to follow:
https://pve.proxmox.com/wiki/PCI(e)_Passthrough
and
A

[TUTORIAL] Thread 'PCI/GPU Passthrough on Proxmox VE 8 : Installation and configuration'

In this article, I propose taking a closer look at the configuration process for setting up PCI Passthrough on Proxmox VE 8.0 (I had initially planned this article for Proxmox VE 7, but since the new version has just been released, it's an opportunity to test!). This article will be the beginning of a series where I'll go into more detail on how to configure different types of VMs (Linux, Windows, macOS and BSD).

I'd also like to thank leesteken for his valuable recommendations and corrections to the first version of this post.

  • PCI/GPU Passthrough on Proxmox VE 8: Windows 10 &...
  • Like

Information about my system:
PVE 8.4.1 running on an old ASUS Prime Z270-K motherboard with an i7-7700 CPU & 96GB RAM
Broadcom 9500-8i as PCIe passthrough to a TrueNAS VM (no issues here)
Intel I219-V NIC PCIe
WD Red 2TB NVME (bootdrive, VM's)
Intel DC P4600 NVME PCIe, which I am trying to passthrough to the TrueNAS VM.

Looking at beneath table the DC P4600 is in its own IOMMU group (15), so I did not expect any issues.
Whereas the 9500-8i is in IOMMU Group 2, which is shared with "6th-10th Gen Core Processor PCIe Controller (x16)", but this causes no issues.

Based on lspci -nnk I created a vfio.conf like this, but it did not help:
options vfio-pci ids=8086:0a55,1028:1fe8
softdep nvme pre: vfio-pci

Although the output looks correct for 03:00.0, with vfio-pci used as the kernel driver.

I am still very much a proxmox beginner, and have exhausted my google abilities, so I would appreciate any help I can get.


lspci -nnk
00:00.0 Host bridge [0600]: Intel Corporation Xeon E3-1200 v6/7th Gen Core Processor Host Bridge/DRAM Registers [8086:591f] (rev 05)
Subsystem: ASUSTeK Computer Inc. Xeon E3-1200 v6/7th Gen Core Processor Host Bridge/DRAM Registers [1043:8694]
Kernel driver in use: skl_uncore
00:01.0 PCI bridge [0604]: Intel Corporation 6th-10th Gen Core Processor PCIe Controller (x16) [8086:1901] (rev 05)
Subsystem: ASUSTeK Computer Inc. 6th-10th Gen Core Processor PCIe Controller (x16) [1043:8694]
Kernel driver in use: pcieport
00:02.0 VGA compatible controller [0300]: Intel Corporation HD Graphics 630 [8086:5912] (rev 04)
DeviceName: Onboard IGD
Subsystem: ASUSTeK Computer Inc. HD Graphics 630 (H110I-PLUS Motherboard) [1043:8694]
Kernel driver in use: i915
Kernel modules: i915
00:14.0 USB controller [0c03]: Intel Corporation 200 Series/Z370 Chipset Family USB 3.0 xHCI Controller [8086:a2af]
Subsystem: ASUSTeK Computer Inc. 200 Series/Z370 Chipset Family USB 3.0 xHCI Controller [1043:8694]
Kernel driver in use: xhci_hcd
Kernel modules: xhci_pci
00:16.0 Communication controller [0780]: Intel Corporation 200 Series PCH CSME HECI #1 [8086:a2ba]
Subsystem: ASUSTeK Computer Inc. 200 Series PCH CSME HECI [1043:8694]
Kernel driver in use: mei_me
Kernel modules: mei_me
00:17.0 SATA controller [0106]: Intel Corporation 200 Series PCH SATA controller [AHCI mode] [8086:a282]
Subsystem: ASUSTeK Computer Inc. 200 Series PCH SATA controller [AHCI mode] [1043:8694]
Kernel driver in use: ahci
Kernel modules: ahci
00:1b.0 PCI bridge [0604]: Intel Corporation 200 Series PCH PCI Express Root Port #17 [8086:a2e7] (rev f0)
Subsystem: ASUSTeK Computer Inc. 200 Series PCH PCI Express Root Port [1043:8694]
Kernel driver in use: pcieport
00:1b.4 PCI bridge [0604]: Intel Corporation 200 Series PCH PCI Express Root Port #21 [8086:a2eb] (rev f0)
Kernel driver in use: pcieport
00:1c.0 PCI bridge [0604]: Intel Corporation 200 Series PCH PCI Express Root Port #1 [8086:a290] (rev f0)
Subsystem: ASUSTeK Computer Inc. 200 Series PCH PCI Express Root Port [1043:8694]
Kernel driver in use: pcieport
00:1c.2 PCI bridge [0604]: Intel Corporation 200 Series PCH PCI Express Root Port #3 [8086:a292] (rev f0)
Subsystem: ASUSTeK Computer Inc. 200 Series PCH PCI Express Root Port [1043:8694]
Kernel driver in use: pcieport
00:1c.4 PCI bridge [0604]: Intel Corporation 200 Series PCH PCI Express Root Port #5 [8086:a294] (rev f0)
Subsystem: ASUSTeK Computer Inc. 200 Series PCH PCI Express Root Port [1043:8694]
Kernel driver in use: pcieport
00:1c.7 PCI bridge [0604]: Intel Corporation 200 Series PCH PCI Express Root Port #8 [8086:a297] (rev f0)
Subsystem: ASUSTeK Computer Inc. 200 Series PCH PCI Express Root Port [1043:8694]
Kernel driver in use: pcieport
00:1d.0 PCI bridge [0604]: Intel Corporation 200 Series PCH PCI Express Root Port #9 [8086:a298] (rev f0)
Subsystem: ASUSTeK Computer Inc. 200 Series PCH PCI Express Root Port [1043:8694]
Kernel driver in use: pcieport
00:1f.0 ISA bridge [0601]: Intel Corporation 200 Series PCH LPC Controller (Z270) [8086:a2c5]
Subsystem: ASUSTeK Computer Inc. 200 Series PCH LPC Controller (Z270) [1043:8694]
00:1f.2 Memory controller [0580]: Intel Corporation 200 Series/Z370 Chipset Family Power Management Controller [8086:a2a1]
Subsystem: ASUSTeK Computer Inc. 200 Series/Z370 Chipset Family Power Management Controller [1043:8694]
00:1f.4 SMBus [0c05]: Intel Corporation 200 Series/Z370 Chipset Family SMBus Controller [8086:a2a3]
Subsystem: ASUSTeK Computer Inc. 200 Series/Z370 Chipset Family SMBus Controller [1043:8694]
Kernel driver in use: i801_smbus
Kernel modules: i2c_i801
00:1f.6 Ethernet controller [0200]: Intel Corporation Ethernet Connection (2) I219-V [8086:15b8]
Subsystem: ASUSTeK Computer Inc. Ethernet Connection (2) I219-V [1043:8672]
Kernel driver in use: e1000e
Kernel modules: e1000e
01:00.0 Serial Attached SCSI controller [0107]: Broadcom / LSI Fusion-MPT 12GSAS/PCIe Secure SAS38xx [1000:00e6]
Subsystem: Broadcom / LSI 9500-8i Tri-Mode HBA [1000:4060]
Kernel driver in use: mpt3sas
Kernel modules: mpt3sas
03:00.0 Non-Volatile memory controller [0108]: Intel Corporation NVMe DC SSD [3DNAND, Beta Rock Controller] [8086:0a55]
Subsystem: Dell Express Flash NVMe 2.0TB HHHL AIC (P4600) [1028:1fe8]
Kernel driver in use: vfio-pci
Kernel modules: nvme
05:00.0 PCI bridge [0604]: ASMedia Technology Inc. ASM1083/1085 PCIe to PCI Bridge [1b21:1080] (rev 04)
Subsystem: ASUSTeK Computer Inc. ASM1083/1085 PCIe to PCI Bridge [1043:8489]
07:00.0 USB controller [0c03]: ASMedia Technology Inc. ASM1142 USB 3.1 Host Controller [1b21:1242]
Subsystem: ASUSTeK Computer Inc. ASM1142 USB 3.1 Host Controller [1043:8675]
Kernel driver in use: xhci_hcd
Kernel modules: xhci_pci
08:00.0 Ethernet controller [0200]: Intel Corporation Ethernet Controller I226-V [8086:125c] (rev 04)
Subsystem: Intel Corporation Ethernet Controller I226-V [8086:0000]
Kernel driver in use: igc
Kernel modules: igc
09:00.0 Non-Volatile memory controller [0108]: Sandisk Corp WD Black SN750 / PC SN730 NVMe SSD [15b7:5006]
Subsystem: Sandisk Corp SanDisk Extreme Pro / WD Black SN750 / PC SN730 / Red SN700 NVMe SSD [15b7:5006]
Kernel driver in use: nvme
Kernel modules: nvme
 
Code: 
pvesh get /nodes/jj/hardware/pci --pci-class-blacklist ""
┌──────────┬────────┬──────────────┬────────────┬────────┬───────────────────────────────────────────────────────────────────┬──────┬──────────────────┬───────────────────────────────────────────┬──────────────────┬──────────────
│ class    │ device │ id           │ iommugroup │ vendor │ device_name                                                       │ mdev │ subsystem_device │ subsystem_device_name                     │ subsystem_vendor │ subsystem_ven
╞══════════╪════════╪══════════════╪════════════╪════════╪═══════════════════════════════════════════════════════════════════╪══════╪══════════════════╪═══════════════════════════════════════════╪══════════════════╪══════════════
│ 0x010601 │ 0xa282 │ 0000:00:17.0 │          5 │ 0x8086 │ 200 Series PCH SATA controller [AHCI mode]                        │      │ 0x8694           │                                           │ 0x1043           │ ASUSTeK Compu
├──────────┼────────┼──────────────┼────────────┼────────┼───────────────────────────────────────────────────────────────────┼──────┼──────────────────┼───────────────────────────────────────────┼──────────────────┼──────────────
│ 0x010700 │ 0x00e6 │ 0000:01:00.0 │          2 │ 0x1000 │ Fusion-MPT 12GSAS/PCIe Secure SAS38xx                             │      │ 0x4060           │ 9500-8i Tri-Mode HBA                      │ 0x1000           │ Broadcom / LS
├──────────┼────────┼──────────────┼────────────┼────────┼───────────────────────────────────────────────────────────────────┼──────┼──────────────────┼───────────────────────────────────────────┼──────────────────┼──────────────
│ 0x010802 │ 0x0a55 │ 0000:03:00.0 │         15 │ 0x8086 │ NVMe DC SSD [3DNAND, Beta Rock Controller]                        │      │ 0x1fe8           │ Express Flash NVMe 2.0TB HHHL AIC (P4600) │ 0x1028           │ Dell         
├──────────┼────────┼──────────────┼────────────┼────────┼───────────────────────────────────────────────────────────────────┼──────┼──────────────────┼───────────────────────────────────────────┼──────────────────┼──────────────
│ 0x010802 │ 0x5006 │ 0000:09:00.0 │         19 │ 0x15b7 │ WD Black SN750 / PC SN730 NVMe SSD                                │      │ 0x5006           │                                           │ 0x15b7           │ Sandisk Corp
├──────────┼────────┼──────────────┼────────────┼────────┼───────────────────────────────────────────────────────────────────┼──────┼──────────────────┼───────────────────────────────────────────┼──────────────────┼──────────────
│ 0x020000 │ 0x15b8 │ 0000:00:1f.6 │         14 │ 0x8086 │ Ethernet Connection (2) I219-V                                    │      │ 0x8672           │                                           │ 0x1043           │ ASUSTeK Compu
├──────────┼────────┼──────────────┼────────────┼────────┼───────────────────────────────────────────────────────────────────┼──────┼──────────────────┼───────────────────────────────────────────┼──────────────────┼──────────────
│ 0x020000 │ 0x125c │ 0000:08:00.0 │         18 │ 0x8086 │ Ethernet Controller I226-V                                        │      │ 0x0000           │                                           │ 0x8086           │ Intel Corpora
├──────────┼────────┼──────────────┼────────────┼────────┼───────────────────────────────────────────────────────────────────┼──────┼──────────────────┼───────────────────────────────────────────┼──────────────────┼──────────────
│ 0x030000 │ 0x5912 │ 0000:00:02.0 │          0 │ 0x8086 │ HD Graphics 630                                                   │      │ 0x8694           │                                           │ 0x1043           │ ASUSTeK Compu
├──────────┼────────┼──────────────┼────────────┼────────┼───────────────────────────────────────────────────────────────────┼──────┼──────────────────┼───────────────────────────────────────────┼──────────────────┼──────────────
│ 0x058000 │ 0xa2a1 │ 0000:00:1f.2 │         13 │ 0x8086 │ 200 Series/Z370 Chipset Family Power Management Controller        │      │ 0x8694           │                                           │ 0x1043           │ ASUSTeK Compu
├──────────┼────────┼──────────────┼────────────┼────────┼───────────────────────────────────────────────────────────────────┼──────┼──────────────────┼───────────────────────────────────────────┼──────────────────┼──────────────
│ 0x060000 │ 0x591f │ 0000:00:00.0 │          1 │ 0x8086 │ Xeon E3-1200 v6/7th Gen Core Processor Host Bridge/DRAM Registers │      │ 0x8694           │                                           │ 0x1043           │ ASUSTeK Compu
├──────────┼────────┼──────────────┼────────────┼────────┼───────────────────────────────────────────────────────────────────┼──────┼──────────────────┼───────────────────────────────────────────┼──────────────────┼──────────────
│ 0x060100 │ 0xa2c5 │ 0000:00:1f.0 │         13 │ 0x8086 │ 200 Series PCH LPC Controller (Z270)                              │      │ 0x8694           │                                           │ 0x1043           │ ASUSTeK Compu
├──────────┼────────┼──────────────┼────────────┼────────┼───────────────────────────────────────────────────────────────────┼──────┼──────────────────┼───────────────────────────────────────────┼──────────────────┼──────────────
│ 0x060400 │ 0x1901 │ 0000:00:01.0 │          2 │ 0x8086 │ 6th-10th Gen Core Processor PCIe Controller (x16)                 │      │ 0x8694           │                                           │ 0x1043           │ ASUSTeK Compu
├──────────┼────────┼──────────────┼────────────┼────────┼───────────────────────────────────────────────────────────────────┼──────┼──────────────────┼───────────────────────────────────────────┼──────────────────┼──────────────
│ 0x060400 │ 0xa2e7 │ 0000:00:1b.0 │          6 │ 0x8086 │ 200 Series PCH PCI Express Root Port #17                          │      │ 0x8694           │                                           │ 0x1043           │ ASUSTeK Compu
├──────────┼────────┼──────────────┼────────────┼────────┼───────────────────────────────────────────────────────────────────┼──────┼──────────────────┼───────────────────────────────────────────┼──────────────────┼──────────────
│ 0x060400 │ 0xa2eb │ 0000:00:1b.4 │          7 │ 0x8086 │ 200 Series PCH PCI Express Root Port #21                          │      │ 0x0000           │                                           │ 0x0000           │             
├──────────┼────────┼──────────────┼────────────┼────────┼───────────────────────────────────────────────────────────────────┼──────┼──────────────────┼───────────────────────────────────────────┼──────────────────┼──────────────
│ 0x060400 │ 0xa290 │ 0000:00:1c.0 │          8 │ 0x8086 │ 200 Series PCH PCI Express Root Port #1                           │      │ 0x8694           │                                           │ 0x1043           │ ASUSTeK Compu
├──────────┼────────┼──────────────┼────────────┼────────┼───────────────────────────────────────────────────────────────────┼──────┼──────────────────┼───────────────────────────────────────────┼──────────────────┼──────────────
│ 0x060400 │ 0xa292 │ 0000:00:1c.2 │          9 │ 0x8086 │ 200 Series PCH PCI Express Root Port #3                           │      │ 0x8694           │                                           │ 0x1043           │ ASUSTeK Compu
├──────────┼────────┼──────────────┼────────────┼────────┼───────────────────────────────────────────────────────────────────┼──────┼──────────────────┼───────────────────────────────────────────┼──────────────────┼──────────────
│ 0x060400 │ 0xa294 │ 0000:00:1c.4 │         10 │ 0x8086 │ 200 Series PCH PCI Express Root Port #5                           │      │ 0x8694           │                                           │ 0x1043           │ ASUSTeK Compu
├──────────┼────────┼──────────────┼────────────┼────────┼───────────────────────────────────────────────────────────────────┼──────┼──────────────────┼───────────────────────────────────────────┼──────────────────┼──────────────
│ 0x060400 │ 0xa297 │ 0000:00:1c.7 │         11 │ 0x8086 │ 200 Series PCH PCI Express Root Port #8                           │      │ 0x8694           │                                           │ 0x1043           │ ASUSTeK Compu
├──────────┼────────┼──────────────┼────────────┼────────┼───────────────────────────────────────────────────────────────────┼──────┼──────────────────┼───────────────────────────────────────────┼──────────────────┼──────────────
│ 0x060400 │ 0xa298 │ 0000:00:1d.0 │         12 │ 0x8086 │ 200 Series PCH PCI Express Root Port #9                           │      │ 0x8694           │                                           │ 0x1043           │ ASUSTeK Compu
├──────────┼────────┼──────────────┼────────────┼────────┼───────────────────────────────────────────────────────────────────┼──────┼──────────────────┼───────────────────────────────────────────┼──────────────────┼──────────────
│ 0x060400 │ 0x1080 │ 0000:05:00.0 │         16 │ 0x1b21 │ ASM1083/1085 PCIe to PCI Bridge                                   │      │ 0x8489           │                                           │ 0x1043           │ ASUSTeK Compu
├──────────┼────────┼──────────────┼────────────┼────────┼───────────────────────────────────────────────────────────────────┼──────┼──────────────────┼───────────────────────────────────────────┼──────────────────┼──────────────
│ 0x078000 │ 0xa2ba │ 0000:00:16.0 │          4 │ 0x8086 │ 200 Series PCH CSME HECI #1                                       │      │ 0x8694           │                                           │ 0x1043           │ ASUSTeK Compu
├──────────┼────────┼──────────────┼────────────┼────────┼───────────────────────────────────────────────────────────────────┼──────┼──────────────────┼───────────────────────────────────────────┼──────────────────┼──────────────
│ 0x0c0330 │ 0xa2af │ 0000:00:14.0 │          3 │ 0x8086 │ 200 Series/Z370 Chipset Family USB 3.0 xHCI Controller            │      │ 0x8694           │                                           │ 0x1043           │ ASUSTeK Compu
├──────────┼────────┼──────────────┼────────────┼────────┼───────────────────────────────────────────────────────────────────┼──────┼──────────────────┼───────────────────────────────────────────┼──────────────────┼──────────────
│ 0x0c0330 │ 0x1242 │ 0000:07:00.0 │         17 │ 0x1b21 │ ASM1142 USB 3.1 Host Controller                                   │      │ 0x8675           │                                           │ 0x1043           │ ASUSTeK Compu
├──────────┼────────┼──────────────┼────────────┼────────┼───────────────────────────────────────────────────────────────────┼──────┼──────────────────┼───────────────────────────────────────────┼──────────────────┼──────────────
│ 0x0c0500 │ 0xa2a3 │ 0000:00:1f.4 │         13 │ 0x8086 │ 200 Series/Z370 Chipset Family SMBus Controller                   │      │ 0x8694           │                                           │ 0x1043           │ ASUSTeK Compu
└──────────┴────────┴──────────────┴────────────┴────────┴───────────────────────────────────────────────────────────────────┴──────┴──────────────────┴───────────────────────────────────────────┴──────────────────┴──────────────
 
It's very much motherboard and device specific. Search for work-arounds for your specific NVMe device or maybe generic issues that people run into with passthrough of NVMe devices. I have no experience with NVMe passthrough and prefer virtual disks (which are much easier to backup).
 
You must log in or register to reply here.
Top Bottom