Problem statement
Having issues passing a legacy PCI bridge card / PMC daughter card component through to a Proxmox Virtual Machine (VM). Using the Proxmox configuration for PCI pass-through shown below. Adding the target PCI device at 3C:04.0 we are unable to start the target VM. Attempting to start the VM causes the Proxmox OS to reboot.
Looking for suggestions on what else to try.
1. Hardware/OS
Server: Dell R740 (Intel processors)
PCI Device: Dynamic Engineering PCIEBPMCX1 PCI-Express to PCI/PCI-X Bridge w/ PMC daughter card.
PMC card (daughter card) is a Power PC: Freescale Semiconductor PCI Device.
OS: Proxmox 8.2.4 (6.8.12-1)
The Power PC Semiconductor PCI device plugged into the PMC connector on the PCIEBPMCX1 is the target for pass-through.
2. Kernel Command Line Options
Attempted to use several combinations of extra command line parameters. Current configuration is shown below from /etc/default/grub
# If you change this file, run 'update-grub' afterwards to update
# /boot/grub/grub.cfg.
# For full documentation of the options in this file, see:
# info -f grub -n 'Simple configuration'
GRUB_DEFAULT=0
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian`
GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt pcie_acs_override=downstream,multifunction initcall_blacklist=
sysfb_init"
GRUB_CMDLINE_LINUX=""
3. VFIO Modules
Loaded required VFIO modules required for Proxmox pass-through:
# /etc/modules: kernel modules to load at boot time.
#
# This file contains the names of kernel modules that should be loaded
# at boot time, one per line. Lines beginning with "#" are ignored.
# Parameters can be specified after the module name.
vfio
vfio_iommu_type1
vfio_pci
Confirmed the VFIO modules are loaded and available:
root@benchserver02:/etc/default# lsmod | grep vfio
vfio_pci 12288 0
vfio_pci_core 81920 1 vfio_pci
irqbypass 12288 2 vfio_pci_core,kvm
vfio_iommu_type1 45056 0
vfio 65536 3 vfio_pci_core,vfio_iommu_type1,vfio_pci
iommufd 94208 1 vfio
4. Confirm IOMMU and IRQ remapping enabled
Output from DMESG at startup confirms that IOMMU and IRQ remapping are enabled:
[ 0.102199] DMAR: IOMMU enabled
[ 0.262929] DMAR-IR: Enabled IRQ remapping in x2apic mode
5. Confirm vfio module kernel driver is selected for target pci device
lspci tree dump shows the motherboard PCI bridge (3a), and the Tundra board (3b), along with the target PMC daughter card at 3c:04.0
| +-1e.4
| +-1e.5
| \-1e.6
+-[0000:3a]-+-00.0-[3b-3c]----00.0-[3c]----04.0
| +-05.0
| +-05.2
| +-05.4
The PCIEBPMCX1 is at 3b:00.0
3b:00.0 PCI bridge: Tundra Semiconductor Corp. Device 8114 (rev 03) (prog-if 00 [Normal decode])
Flags: bus master, fast devsel, latency 0, NUMA node 0, IOMMU group 1
Bus: primary=3b, secondary=3c, subordinate=3c, sec-latency=0
I/O behind bridge: [disabled] [32-bit]
Memory behind bridge: ab100000-ab1fffff [size=1M] [32-bit]
Prefetchable memory behind bridge: ab000000-ab0fffff [size=1M] [32-bit]
Capabilities: [80] PCI-X bridge device
Capabilities: [a0] Power Management version 3
Capabilities: [c0] Express PCI-Express to PCI/PCI-X Bridge, MSI 00
Capabilities: [100] Advanced Error Reporting
lspci dump of 3c:04.0 shows that the PMC daughter board PCI device is using the VFIO pass-through kernel driver:
3c:04.0 Power PC: Freescale Semiconductor Inc Device XXXX (rev 21)
Flags: bus master, 66MHz, fast devsel, latency 128, IRQ 255, NUMA node 0, IOMMU group 1
Memory at ab100000 (32-bit, non-prefetchable) [size=1M]
Memory at ab000000 (32-bit, prefetchable) [size=8K]
Capabilities: [48] CompactPCI hot-swap <?>
Capabilities: [80] Power Management version 3
Kernel driver in use: vfio-pci
6. IOMMU group layout
Dell R740 mother PCI bridge is in group 0:
IOMMU Group 0:
3a:00.0 PCI bridge [0604]: Intel Corporation Sky Lake-E PCI Express Root Port A [8086:2030] (rev 07)
The PCIEBPMCX1 PCI bridge card w/ PMC daughter card are in group 1:
IOMMU Group 1:
3b:00.0 PCI bridge [0604]: Tundra Semiconductor Corp. Device [10e3:8114] (rev 03)
3c:04.0 Power PC [0b20]: Freescale Semiconductor Inc Device [XXXX:XXXX] (rev 21)
7. Passthru Options
The conf file which contains the vfio options to add the target device for passthru:
From /etc/modprobe.d/vfio.conf:
options vfio-pci ids=10e3:8114,XXXX:XXXX disable_idle_d3=1
Note: this was tried with and without the Tundra bridge device ID. XXXX:XXXX is the device ID for the target PCI device.
8. Passthru cmdline cmd to add pci device to VM 101
qm set 101 -hostpci0 3c:04.0,pcie=0
Note: this was tried with pcie=0 and 1.
Having issues passing a legacy PCI bridge card / PMC daughter card component through to a Proxmox Virtual Machine (VM). Using the Proxmox configuration for PCI pass-through shown below. Adding the target PCI device at 3C:04.0 we are unable to start the target VM. Attempting to start the VM causes the Proxmox OS to reboot.
Looking for suggestions on what else to try.
1. Hardware/OS
Server: Dell R740 (Intel processors)
PCI Device: Dynamic Engineering PCIEBPMCX1 PCI-Express to PCI/PCI-X Bridge w/ PMC daughter card.
PMC card (daughter card) is a Power PC: Freescale Semiconductor PCI Device.
OS: Proxmox 8.2.4 (6.8.12-1)
The Power PC Semiconductor PCI device plugged into the PMC connector on the PCIEBPMCX1 is the target for pass-through.
2. Kernel Command Line Options
Attempted to use several combinations of extra command line parameters. Current configuration is shown below from /etc/default/grub
# If you change this file, run 'update-grub' afterwards to update
# /boot/grub/grub.cfg.
# For full documentation of the options in this file, see:
# info -f grub -n 'Simple configuration'
GRUB_DEFAULT=0
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian`
GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt pcie_acs_override=downstream,multifunction initcall_blacklist=
sysfb_init"
GRUB_CMDLINE_LINUX=""
3. VFIO Modules
Loaded required VFIO modules required for Proxmox pass-through:
# /etc/modules: kernel modules to load at boot time.
#
# This file contains the names of kernel modules that should be loaded
# at boot time, one per line. Lines beginning with "#" are ignored.
# Parameters can be specified after the module name.
vfio
vfio_iommu_type1
vfio_pci
Confirmed the VFIO modules are loaded and available:
root@benchserver02:/etc/default# lsmod | grep vfio
vfio_pci 12288 0
vfio_pci_core 81920 1 vfio_pci
irqbypass 12288 2 vfio_pci_core,kvm
vfio_iommu_type1 45056 0
vfio 65536 3 vfio_pci_core,vfio_iommu_type1,vfio_pci
iommufd 94208 1 vfio
4. Confirm IOMMU and IRQ remapping enabled
Output from DMESG at startup confirms that IOMMU and IRQ remapping are enabled:
[ 0.102199] DMAR: IOMMU enabled
[ 0.262929] DMAR-IR: Enabled IRQ remapping in x2apic mode
5. Confirm vfio module kernel driver is selected for target pci device
lspci tree dump shows the motherboard PCI bridge (3a), and the Tundra board (3b), along with the target PMC daughter card at 3c:04.0
| +-1e.4
| +-1e.5
| \-1e.6
+-[0000:3a]-+-00.0-[3b-3c]----00.0-[3c]----04.0
| +-05.0
| +-05.2
| +-05.4
The PCIEBPMCX1 is at 3b:00.0
3b:00.0 PCI bridge: Tundra Semiconductor Corp. Device 8114 (rev 03) (prog-if 00 [Normal decode])
Flags: bus master, fast devsel, latency 0, NUMA node 0, IOMMU group 1
Bus: primary=3b, secondary=3c, subordinate=3c, sec-latency=0
I/O behind bridge: [disabled] [32-bit]
Memory behind bridge: ab100000-ab1fffff [size=1M] [32-bit]
Prefetchable memory behind bridge: ab000000-ab0fffff [size=1M] [32-bit]
Capabilities: [80] PCI-X bridge device
Capabilities: [a0] Power Management version 3
Capabilities: [c0] Express PCI-Express to PCI/PCI-X Bridge, MSI 00
Capabilities: [100] Advanced Error Reporting
lspci dump of 3c:04.0 shows that the PMC daughter board PCI device is using the VFIO pass-through kernel driver:
3c:04.0 Power PC: Freescale Semiconductor Inc Device XXXX (rev 21)
Flags: bus master, 66MHz, fast devsel, latency 128, IRQ 255, NUMA node 0, IOMMU group 1
Memory at ab100000 (32-bit, non-prefetchable) [size=1M]
Memory at ab000000 (32-bit, prefetchable) [size=8K]
Capabilities: [48] CompactPCI hot-swap <?>
Capabilities: [80] Power Management version 3
Kernel driver in use: vfio-pci
6. IOMMU group layout
Dell R740 mother PCI bridge is in group 0:
IOMMU Group 0:
3a:00.0 PCI bridge [0604]: Intel Corporation Sky Lake-E PCI Express Root Port A [8086:2030] (rev 07)
The PCIEBPMCX1 PCI bridge card w/ PMC daughter card are in group 1:
IOMMU Group 1:
3b:00.0 PCI bridge [0604]: Tundra Semiconductor Corp. Device [10e3:8114] (rev 03)
3c:04.0 Power PC [0b20]: Freescale Semiconductor Inc Device [XXXX:XXXX] (rev 21)
7. Passthru Options
The conf file which contains the vfio options to add the target device for passthru:
From /etc/modprobe.d/vfio.conf:
options vfio-pci ids=10e3:8114,XXXX:XXXX disable_idle_d3=1
Note: this was tried with and without the Tundra bridge device ID. XXXX:XXXX is the device ID for the target PCI device.
8. Passthru cmdline cmd to add pci device to VM 101
qm set 101 -hostpci0 3c:04.0,pcie=0
Note: this was tried with pcie=0 and 1.
