[SOLVED] Pcie Passthrough on PowerEdge R740xd

naustdal

New Member
Apr 30, 2025
2
0
1
I struggle a bit with PCIE passthrough and would appreciate some help and guiding.

My hardware is 24-bay Dell PowerEdge R740xd.

I have search on forum and there is a lot of similar challenge as what I struggle with. But I really don’t catch it. Doesn’t understand how to implement host blacklist.
Its simple and I know it, sorry for asking…

My 2 main disk controller i want in VM100:
Code:
5f:04.0 PCI bridge: PLX Technology, Inc. PEX 9733 33-lane, 9-port PCI Express Gen 3 (8.0 GT/s) Switch (rev b0) (prog-if 00 [Normal decode])
        Subsystem: Dell PEX 9733 33-lane, 9-port PCI Express Gen 3 (8.0 GT/s) Switch
        Flags: bus master, fast devsel, latency 0, IRQ 33, NUMA node 0, IOMMU group 2
        Bus: primary=5f, secondary=60, subordinate=6d, sec-latency=0
        I/O behind bridge: [disabled] [32-bit]
        Memory behind bridge: b8800000-b94fffff [size=13M] [32-bit]
        Prefetchable memory behind bridge: 38c000000000-38c0023fffff [size=36M] [32-bit]
        Capabilities: [40] Power Management version 3
        Capabilities: [48] MSI: Enable+ Count=1/8 Maskable+ 64bit+
        Capabilities: [68] Express Downstream Port (Slot+), MSI 00
        Capabilities: [a4] Subsystem: Dell PEX 9733 33-lane, 9-port PCI Express Gen 3 (8.0 GT/s) Switch
        Capabilities: [100] Device Serial Number 00-0e-df-10-b5-97-00-b0
        Capabilities: [fb4] Advanced Error Reporting
        Capabilities: [138] Power Budgeting <?>
        Capabilities: [10c] Secondary PCI Express
        Capabilities: [f24] Access Control Services
        Capabilities: [b70] Vendor Specific Information: ID=0001 Rev=0 Len=010 <?>
        Kernel driver in use: pcieport

5f:04.0 PCI bridge: PLX Technology, Inc. PEX 9733 33-lane, 9-port PCI Express Gen 3 (8.0 GT/s) Switch (rev b0) (prog-if 00 [Normal decode])
        Subsystem: Dell PEX 9733 33-lane, 9-port PCI Express Gen 3 (8.0 GT/s) Switch
        Flags: bus master, fast devsel, latency 0, IRQ 33, NUMA node 0, IOMMU group 2
        Bus: primary=5f, secondary=60, subordinate=6d, sec-latency=0
        I/O behind bridge: [disabled] [32-bit]
        Memory behind bridge: b8800000-b94fffff [size=13M] [32-bit]
        Prefetchable memory behind bridge: 38c000000000-38c0023fffff [size=36M] [32-bit]
        Capabilities: [40] Power Management version 3
        Capabilities: [48] MSI: Enable+ Count=1/8 Maskable+ 64bit+
        Capabilities: [68] Express Downstream Port (Slot+), MSI 00
        Capabilities: [a4] Subsystem: Dell PEX 9733 33-lane, 9-port PCI Express Gen 3 (8.0 GT/s) Switch
        Capabilities: [100] Device Serial Number 00-0e-df-10-b5-97-00-b0
        Capabilities: [fb4] Advanced Error Reporting
        Capabilities: [138] Power Budgeting <?>
        Capabilities: [10c] Secondary PCI Express
        Capabilities: [f24] Access Control Services
        Capabilities: [b70] Vendor Specific Information: ID=0001 Rev=0 Len=010 <?>
        Kernel driver in use: pcieport

Additional, i have an 1 pcie card with 4 NVMe m2 card. Thats cards show up in 4 diffrent output. I would like to merge that to 1 output:
Code:
af:00.0 Non-Volatile memory controller: Kingston Technology Company, Inc. NV3 NVMe SSD E27T (DRAM-less) (rev 01) (prog-if 02 [NVM Express])
        Subsystem: Kingston Technology Company, Inc. NV3 NVMe SSD E27T (DRAM-less)
        Flags: bus master, fast devsel, latency 0, IRQ 279, NUMA node 1, IOMMU group 42
        Memory at e1300000 (64-bit, non-prefetchable) [size=16K]
        Capabilities: [80] Express Endpoint, MSI 00
        Capabilities: [d0] MSI-X: Enable+ Count=33 Masked-
        Capabilities: [e0] MSI: Enable- Count=1/32 Maskable+ 64bit+
        Capabilities: [f8] Power Management version 3
        Capabilities: [100] Latency Tolerance Reporting
        Capabilities: [110] L1 PM Substates
        Capabilities: [128] Alternative Routing-ID Interpretation (ARI)
        Capabilities: [1e0] Data Link Feature <?>
        Capabilities: [200] Advanced Error Reporting
        Capabilities: [2a0] Power Budgeting <?>
        Capabilities: [300] Secondary PCI Express
        Capabilities: [340] Physical Layer 16.0 GT/s <?>
        Capabilities: [378] Lane Margining at the Receiver <?>
        Kernel driver in use: nvme
        Kernel modules: nvme

b0:00.0 Non-Volatile memory controller: Kingston Technology Company, Inc. NV3 NVMe SSD E27T (DRAM-less) (rev 01) (prog-if 02 [NVM Express])
        Subsystem: Kingston Technology Company, Inc. NV3 NVMe SSD E27T (DRAM-less)
        Flags: bus master, fast devsel, latency 0, IRQ 283, NUMA node 1, IOMMU group 43
        Memory at e1200000 (64-bit, non-prefetchable) [size=16K]
        Capabilities: [80] Express Endpoint, MSI 00
        Capabilities: [d0] MSI-X: Enable+ Count=33 Masked-
        Capabilities: [e0] MSI: Enable- Count=1/32 Maskable+ 64bit+
        Capabilities: [f8] Power Management version 3
        Capabilities: [100] Latency Tolerance Reporting
        Capabilities: [110] L1 PM Substates
        Capabilities: [128] Alternative Routing-ID Interpretation (ARI)
        Capabilities: [1e0] Data Link Feature <?>
        Capabilities: [200] Advanced Error Reporting
        Capabilities: [2a0] Power Budgeting <?>
        Capabilities: [300] Secondary PCI Express
        Capabilities: [340] Physical Layer 16.0 GT/s <?>
        Capabilities: [378] Lane Margining at the Receiver <?>
        Kernel driver in use: nvme
        Kernel modules: nvme

b1:00.0 Non-Volatile memory controller: Kingston Technology Company, Inc. NV3 NVMe SSD E27T (DRAM-less) (rev 01) (prog-if 02 [NVM Express])
        Subsystem: Kingston Technology Company, Inc. NV3 NVMe SSD E27T (DRAM-less)
        Flags: bus master, fast devsel, latency 0, IRQ 285, NUMA node 1, IOMMU group 44
        Memory at e1100000 (64-bit, non-prefetchable) [size=16K]
        Capabilities: [80] Express Endpoint, MSI 00
        Capabilities: [d0] MSI-X: Enable+ Count=33 Masked-
        Capabilities: [e0] MSI: Enable- Count=1/32 Maskable+ 64bit+
        Capabilities: [f8] Power Management version 3
        Capabilities: [100] Latency Tolerance Reporting
        Capabilities: [110] L1 PM Substates
        Capabilities: [128] Alternative Routing-ID Interpretation (ARI)
        Capabilities: [1e0] Data Link Feature <?>
        Capabilities: [200] Advanced Error Reporting
        Capabilities: [2a0] Power Budgeting <?>
        Capabilities: [300] Secondary PCI Express
        Capabilities: [340] Physical Layer 16.0 GT/s <?>
        Capabilities: [378] Lane Margining at the Receiver <?>
        Kernel driver in use: nvme
        Kernel modules: nvme

b2:00.0 Non-Volatile memory controller: Kingston Technology Company, Inc. NV3 NVMe SSD E27T (DRAM-less) (rev 01) (prog-if 02 [NVM Express])
        Subsystem: Kingston Technology Company, Inc. NV3 NVMe SSD E27T (DRAM-less)
        Flags: bus master, fast devsel, latency 0, IRQ 288, NUMA node 1, IOMMU group 45
        Memory at e1000000 (64-bit, non-prefetchable) [size=16K]
        Capabilities: [80] Express Endpoint, MSI 00
        Capabilities: [d0] MSI-X: Enable+ Count=33 Masked-
        Capabilities: [e0] MSI: Enable- Count=1/32 Maskable+ 64bit+
        Capabilities: [f8] Power Management version 3
        Capabilities: [100] Latency Tolerance Reporting
        Capabilities: [110] L1 PM Substates
        Capabilities: [128] Alternative Routing-ID Interpretation (ARI)
        Capabilities: [1e0] Data Link Feature <?>
        Capabilities: [200] Advanced Error Reporting
        Capabilities: [2a0] Power Budgeting <?>
        Capabilities: [300] Secondary PCI Express
        Capabilities: [340] Physical Layer 16.0 GT/s <?>
        Capabilities: [378] Lane Margining at the Receiver <?>
        Kernel driver in use: nvme
        Kernel modules: nvme

Would appreciate some help to block those from host and be able to assign it to VM100.
 
Has been tested some different approaches, but its hasn’t lead not any good.
Added "options vfio-pci ids=10b5:9733" in vfio.conf, still got error when starting VM.
Added "blacklist vfio-pci ids=5f:04.0" in blacklist.conf, then it disappears from the list.
Added it manually "qm set 100 -hostpci0 5f:04.0", got error.

Resource Mapping the pci device true web gui and added it true Hardware to VM, got error.
Code:
error writing '1' to '/sys/bus/pci/devices/0000:5f:04.0/reset': Inappropriate ioctl for device
failed to reset PCI device '0000:5f:04.0', but trying to continue as not all devices need a reset
kvm: -device vfio-pci,host=0000:5f:04.0,id=hostpci0,bus=pci.0,addr=0x10: vfio 0000:5f:04.0: Could not open '/dev/vfio/2': No such file or directory
TASK ERROR: start failed: QEMU exited with code 1

After some searching, ended up to focus on Vendor Reset…

Code:
dmesg | grep reset
[   26.542672] pcieport 0000:d7:00.0: unlocked secondary bus reset via: __pci_reset_function_locked+0x3a/0x60
modprobe -v vendor-reset
modprobe: FATAL: Module vendor-reset not found in directory /lib/modules/6.12.25-x64v3-xanmod1

Did an apt update/upgrade, and got follow errors:
Code:
dkms: autoinstall for kernel: 6.12.25-x64v3-xanmod1 failed!
run-parts: /etc/kernel/postinst.d/dkms exited with return code 11
update-initramfs: Generating /boot/initrd.img-6.12.25-x64v3-xanmod1
Running hook script 'zz-proxmox-boot'..
Re-executing '/etc/kernel/postinst.d/zz-proxmox-boot' in new private mount namespace..
No /etc/kernel/proxmox-boot-uuids found, skipping ESP sync.
Re-executing '/etc/kernel/postinst.d/zz-proxmox-boot' in new private mount namespace..
No /etc/kernel/proxmox-boot-uuids found, skipping ESP sync.
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-6.12.25-x64v3-xanmod1
Found initrd image: /boot/initrd.img-6.12.25-x64v3-xanmod1
Found linux image: /boot/vmlinuz-6.8.12-10-pve
Found initrd image: /boot/initrd.img-6.8.12-10-pve
Found linux image: /boot/vmlinuz-6.8.12-9-pve
Found initrd image: /boot/initrd.img-6.8.12-9-pve
Found linux image: /boot/vmlinuz-6.8.12-4-pve
Found initrd image: /boot/initrd.img-6.8.12-4-pve
Found memtest86+ 64bit EFI image: /boot/memtest86+x64.efi
done
dpkg: error processing package linux-image-6.12.25-x64v3-xanmod1 (--configure):
 installed linux-image-6.12.25-x64v3-xanmod1 package post-installation script subprocess returned error exit status 1
dpkg: dependency problems prevent configuration of linux-xanmod-lts-x64v3:
 linux-xanmod-lts-x64v3 depends on linux-image-6.12.25-x64v3-xanmod1; however:
  Package linux-image-6.12.25-x64v3-xanmod1 is not configured yet.

dpkg: error processing package linux-xanmod-lts-x64v3 (--configure):
 dependency problems - leaving unconfigured
Errors were encountered while processing:
 linux-image-6.12.25-x64v3-xanmod1
 linux-xanmod-lts-x64v3
E: Sub-process /usr/bin/dpkg returned an error code (1)
W: Operation was interrupted before it could finis

Not sure how to deal with this.