Pcie/Thunderbolt 4/USB4 Passthrough on Proxmox VE 8.2

S

ss4sgoku

New Member
Jan 17, 2024
5
0
1
hi, this is my second post im a newbie in proxmox so im trying to figure out how to acomplish a full passthrough of my thunderbolt4 that lookslike its conencted via pcie port that port share conection with the network card
First, the system has a Motherboard MSI Z590 ACE with a i5-11600k it has a thunderbolt 4 with intel jhl8540 via pcie slot, Proxmox 8.2 full updated

This is the diagram of the Motherboard z590 ace
and this is the thunderbolt module i have this pictures from this web


This is my grub right now
Code: 
GRUB_DEFAULT=0
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian`
GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on,relax_rmrr iommu=pt intermap=no_x2ap  pcie_port_pm=off pcie_aspm.policy=performance"
GRUB_CMDLINE_LINUX=""

cat /etc/modules
Code: 
vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd
thunderbolt

dmesg | grep 'remapping'
Code: 
[    0.372963] DMAR-IR: Queued invalidation will be enabled to support x2apic and Intr-remapping.
[    0.374575] DMAR-IR: Enabled IRQ remapping in x2apic mode

dmesg | grep 'Intel-IOMMU'
Code: 
[    0.155276] DMAR: Intel-IOMMU: assuming all RMRRs are relaxable. This can lead to instability or data loss
ç

cat /proc/modules | grep pci
Code: 
snd_sof_pci_intel_tgl 12288 0 - Live 0xffffffffc22a3000
snd_sof_intel_hda_common 208896 1 snd_sof_pci_intel_tgl, Live 0xffffffffc225c000
snd_sof_pci 24576 2 snd_sof_pci_intel_tgl,snd_sof_intel_hda_common, Live 0xffffffffc21fb000
snd_sof 356352 3 snd_sof_intel_hda_common,snd_sof_intel_hda,snd_sof_pci, Live 0xffffffffc217f000
snd_soc_acpi_intel_match 102400 2 snd_sof_pci_intel_tgl,snd_sof_intel_hda_common, Live 0xffffffffc2132000
xhci_pci 24576 0 - Live 0xffffffffc04c0000
xhci_pci_renesas 16384 1 xhci_pci, Live 0xffffffffc046a000
spi_intel_pci 12288 0 - Live 0xffffffffc066f000
xhci_hcd 364544 1 xhci_pci, Live 0xffffffffc0599000
spi_intel 32768 1 spi_intel_pci, Live 0xffffffffc0528000
vfio_pci 16384 0 - Live 0xffffffffc049b000
vfio_pci_core 86016 1 vfio_pci, Live 0xffffffffc0478000
irqbypass 12288 2 kvm,vfio_pci_core, Live 0xffffffffc0470000
vfio 69632 3 vfio_pci,vfio_pci_core,vfio_iommu_type1, Live 0xffffffffc043a000

cat /proc/modules | grep usb
Code: 
snd_usb_audio 442368 1 - Live 0xffffffffc1d43000
snd_usbmidi_lib 49152 1 snd_usb_audio, Live 0xffffffffc1d0b000
snd_ump 40960 1 snd_usb_audio, Live 0xffffffffc1cef000
snd_rawmidi 53248 2 snd_usbmidi_lib,snd_ump, Live 0xffffffffc1c76000
mc 86016 1 snd_usb_audio, Live 0xffffffffc1acb000
snd_hwdep 20480 2 snd_usb_audio,snd_hda_codec, Live 0xffffffffc1a8e000
snd_pcm 188416 12 snd_hda_codec_hdmi,snd_sof_intel_hda_common,soundwire_intel,snd_sof,snd_sof_utils,snd_soc_core,snd_compress,snd_pcm_dmaengine,snd_hda_intel,snd_usb_audio,snd_hda_codec,snd_hda_core, Live 0xffffffffc1b4c000
snd 139264 21 snd_seq,snd_hda_codec_hdmi,snd_sof,snd_soc_core,snd_compress,snd_hda_intel,snd_usb_audio,snd_usbmidi_lib,snd_ump,snd_hda_codec,snd_rawmidi,snd_seq_device,snd_hwdep,snd_pcm,snd_timer, Live 0xffffffffc1a9e000
usbmouse 12288 0 - Live 0xffffffffc0b04000
usbhid 69632 0 - Live 0xffffffffc0aed000
hid 167936 2 hid_generic,usbhid, Live 0xffffffffc0ab8000

pvesh get /nodes/pve/hardware/pci --pci-class-blacklist ""
Code: 
│ class    │ device │ id           │ iommugroup │ vendor │ device_name                                        │ mdev │ subsystem_device │ subsystem_device_name │ subsystem_vendor │ subsystem_vendor_name       
╞══════════╪════════╪══════════════╪════════════╪════════╪════════════════════════════════════════════════════╪══════╪══════════════════╪═══════════════════════╪══════════════════╪═════════════════════════════
│ 0x010601 │ 0x43d2 │ 0000:00:17.0 │          7 │ 0x8086 │                                                    │      │ 0x7d04           │                       │ 0x1462           │ Micro-Star International Co.
├──────────┼────────┼──────────────┼────────────┼────────┼────────────────────────────────────────────────────┼──────┼──────────────────┼───────────────────────┼──────────────────┼─────────────────────────────
│ 0x010802 │ 0x5018 │ 0000:02:00.0 │         12 │ 0x1bb1 │ FireCuda 530 SSD                                   │      │ 0x5018           │                       │ 0x1bb1           │ Seagate Technology PLC     
├──────────┼────────┼──────────────┼────────────┼────────┼────────────────────────────────────────────────────┼──────┼──────────────────┼───────────────────────┼──────────────────┼─────────────────────────────
│ 0x018000 │ 0x0018 │ 0000:01:00.0 │         11 │ 0xace1 │                                                    │      │ 0x0018           │                       │ 0xace1           │                             
├──────────┼────────┼──────────────┼────────────┼────────┼────────────────────────────────────────────────────┼──────┼──────────────────┼───────────────────────┼──────────────────┼─────────────────────────────
│ 0x020000 │ 0x15f3 │ 0000:6e:00.0 │         20 │ 0x8086 │ Ethernet Controller I225-V                         │      │ 0x7d04           │                       │ 0x1462           │ Micro-Star International Co.
├──────────┼────────┼──────────────┼────────────┼────────┼────────────────────────────────────────────────────┼──────┼──────────────────┼───────────────────────┼──────────────────┼─────────────────────────────
│ 0x030000 │ 0x4c8a │ 0000:00:02.0 │          0 │ 0x8086 │ RocketLake-S GT1 [UHD Graphics 750]                │      │ 0x7d04           │                       │ 0x1462           │ Micro-Star International Co.
├──────────┼────────┼──────────────┼────────────┼────────┼────────────────────────────────────────────────────┼──────┼──────────────────┼───────────────────────┼──────────────────┼─────────────────────────────
│ 0x040300 │ 0x43c8 │ 0000:00:1f.3 │         10 │ 0x8086 │ Tiger Lake-H HD Audio Controller                   │      │ 0x7d04           │                       │ 0x1462           │ Micro-Star International Co.
├──────────┼────────┼──────────────┼────────────┼────────┼────────────────────────────────────────────────────┼──────┼──────────────────┼───────────────────────┼──────────────────┼─────────────────────────────
│ 0x050000 │ 0x43ef │ 0000:00:14.2 │          5 │ 0x8086 │ Tiger Lake-H Shared SRAM                           │      │ 0x0000           │                       │ 0x0000           │                             
├──────────┼────────┼──────────────┼────────────┼────────┼────────────────────────────────────────────────────┼──────┼──────────────────┼───────────────────────┼──────────────────┼─────────────────────────────
│ 0x060000 │ 0x4c53 │ 0000:00:00.0 │          1 │ 0x8086 │                                                    │      │ 0x7d04           │                       │ 0x1462           │ Micro-Star International Co.
├──────────┼────────┼──────────────┼────────────┼────────┼────────────────────────────────────────────────────┼──────┼──────────────────┼───────────────────────┼──────────────────┼─────────────────────────────
│ 0x060100 │ 0x4385 │ 0000:00:1f.0 │         10 │ 0x8086 │                                                    │      │ 0x7d04           │                       │ 0x1462           │ Micro-Star International Co.
├──────────┼────────┼──────────────┼────────────┼────────┼────────────────────────────────────────────────────┼──────┼──────────────────┼───────────────────────┼──────────────────┼─────────────────────────────
│ 0x060400 │ 0x4c01 │ 0000:00:01.0 │          2 │ 0x8086 │                                                    │      │ 0x7d04           │                       │ 0x1462           │ Micro-Star International Co.
├──────────┼────────┼──────────────┼────────────┼────────┼────────────────────────────────────────────────────┼──────┼──────────────────┼───────────────────────┼──────────────────┼─────────────────────────────
│ 0x060400 │ 0x4c09 │ 0000:00:06.0 │          3 │ 0x8086 │                                                    │      │ 0x0000           │                       │ 0x0000           │                             
├──────────┼────────┼──────────────┼────────────┼────────┼────────────────────────────────────────────────────┼──────┼──────────────────┼───────────────────────┼──────────────────┼─────────────────────────────
│ 0x060400 │ 0x43b8 │ 0000:00:1c.0 │          8 │ 0x8086 │                                                    │      │ 0x7d04           │                       │ 0x1462           │ Micro-Star International Co.
├──────────┼────────┼──────────────┼────────────┼────────┼────────────────────────────────────────────────────┼──────┼──────────────────┼───────────────────────┼──────────────────┼─────────────────────────────
│ 0x060400 │ 0x43bc │ 0000:00:1c.4 │          9 │ 0x8086 │ Tiger Lake-H PCI Express Root Port #5              │      │ 0x7d04           │                       │ 0x1462           │ Micro-Star International Co.
├──────────┼────────┼──────────────┼────────────┼────────┼────────────────────────────────────────────────────┼──────┼──────────────────┼───────────────────────┼──────────────────┼─────────────────────────────
│ 0x060400 │ 0x1136 │ 0000:03:00.0 │         13 │ 0x8086 │ Thunderbolt 4 Bridge [Maple Ridge 4C 2020]         │      │ 0x0000           │                       │ 0x8086           │ Intel Corporation           
├──────────┼────────┼──────────────┼────────────┼────────┼────────────────────────────────────────────────────┼──────┼──────────────────┼───────────────────────┼──────────────────┼─────────────────────────────
│ 0x060400 │ 0x1136 │ 0000:04:00.0 │         14 │ 0x8086 │ Thunderbolt 4 Bridge [Maple Ridge 4C 2020]         │      │ 0x0000           │                       │ 0x8086           │ Intel Corporation           
├──────────┼────────┼──────────────┼────────────┼────────┼────────────────────────────────────────────────────┼──────┼──────────────────┼───────────────────────┼──────────────────┼─────────────────────────────
│ 0x060400 │ 0x1136 │ 0000:04:01.0 │         15 │ 0x8086 │ Thunderbolt 4 Bridge [Maple Ridge 4C 2020]         │      │ 0x0000           │                       │ 0x8086           │ Intel Corporation           
├──────────┼────────┼──────────────┼────────────┼────────┼────────────────────────────────────────────────────┼──────┼──────────────────┼───────────────────────┼──────────────────┼─────────────────────────────
│ 0x060400 │ 0x1136 │ 0000:04:02.0 │         16 │ 0x8086 │ Thunderbolt 4 Bridge [Maple Ridge 4C 2020]         │      │ 0x0000           │                       │ 0x8086           │ Intel Corporation           
├──────────┼────────┼──────────────┼────────────┼────────┼────────────────────────────────────────────────────┼──────┼──────────────────┼───────────────────────┼──────────────────┼─────────────────────────────
│ 0x060400 │ 0x1136 │ 0000:04:03.0 │         17 │ 0x8086 │ Thunderbolt 4 Bridge [Maple Ridge 4C 2020]         │      │ 0x0000           │                       │ 0x8086           │ Intel Corporation           
├──────────┼────────┼──────────────┼────────────┼────────┼────────────────────────────────────────────────────┼──────┼──────────────────┼───────────────────────┼──────────────────┼─────────────────────────────
│ 0x078000 │ 0x43e0 │ 0000:00:16.0 │          6 │ 0x8086 │ Tiger Lake-H Management Engine Interface           │      │ 0x7d04           │                       │ 0x1462           │ Micro-Star International Co.
├──────────┼────────┼──────────────┼────────────┼────────┼────────────────────────────────────────────────────┼──────┼──────────────────┼───────────────────────┼──────────────────┼─────────────────────────────
│ 0x088000 │ 0x4c11 │ 0000:00:08.0 │          4 │ 0x8086 │                                                    │      │ 0x7d04           │                       │ 0x1462           │ Micro-Star International Co.
├──────────┼────────┼──────────────┼────────────┼────────┼────────────────────────────────────────────────────┼──────┼──────────────────┼───────────────────────┼──────────────────┼─────────────────────────────
│ 0x0c0330 │ 0x43ed │ 0000:00:14.0 │          5 │ 0x8086 │ Tiger Lake-H USB 3.2 Gen 2x1 xHCI Host Controller  │      │ 0x7d04           │                       │ 0x1462           │ Micro-Star International Co.
├──────────┼────────┼──────────────┼────────────┼────────┼────────────────────────────────────────────────────┼──────┼──────────────────┼───────────────────────┼──────────────────┼─────────────────────────────
│ 0x0c0330 │ 0x1138 │ 0000:07:00.0 │         19 │ 0x8086 │ Thunderbolt 4 USB Controller [Maple Ridge 4C 2020] │      │ 0x0000           │                       │ 0x0000           │                             
├──────────┼────────┼──────────────┼────────────┼────────┼────────────────────────────────────────────────────┼──────┼──────────────────┼───────────────────────┼──────────────────┼─────────────────────────────
│ 0x0c0340 │ 0x1137 │ 0000:05:00.0 │         18 │ 0x8086 │ Thunderbolt 4 NHI [Maple Ridge 4C 2020]            │      │ 0x0000           │                       │ 0x8086           │ Intel Corporation           
├──────────┼────────┼──────────────┼────────────┼────────┼────────────────────────────────────────────────────┼──────┼──────────────────┼───────────────────────┼──────────────────┼─────────────────────────────
│ 0x0c0500 │ 0x43a3 │ 0000:00:1f.4 │         10 │ 0x8086 │ Tiger Lake-H SMBus Controller                      │      │ 0x7d04           │                       │ 0x1462           │ Micro-Star International Co.
├──────────┼────────┼──────────────┼────────────┼────────┼────────────────────────────────────────────────────┼──────┼──────────────────┼───────────────────────┼──────────────────┼─────────────────────────────
│ 0x0c8000 │ 0x43a4 │ 0000:00:1f.5 │         10 │ 0x8086 │ Tiger Lake-H SPI Controller                        │      │ 0x7d04           │                       │ 0x1462           │ Micro-Star International Co.
└──────────┴────────┴──────────────┴────────────┴────────┴────────────────────────────────────────────────────┴──────┴──────────────────┴───────────────────────┴──────────────────┴─────────────────────────────
 
/etc/modprobe.d/vfio.conf
Code: 
options vfio-pci ids=8086:43b8,8086:1136,8086:1137,8086:1138
softdep thunderbolt pre: vfio-pci
softdep pcieport pre: vfio-pci
softdep xhci_pci pre: vfio-pci
softdep xhci_hcd pre: vfio-pci

/etc/modprobe.d/blacklist.conf
Code: 
blacklist pcieport
blacklist xhci_hcd
blacklist thunderbolt

/etc/modprobe.d/kvm.conf
Code: 
options kvm ignore_msrs=1

/etc/modprobe.d/iommu_unsafe_interrupts.conf
Code: 
options vfio_iommu_type1 allow_unsafe_interrupts=1

dmesg | grep -e DMAR -e IOMMU
Code: 
[    0.016187] ACPI: DMAR 0x0000000074ADC000 000088 (v02 INTEL  EDK2     00000002      01000013)
[    0.016215] ACPI: Reserving DMAR table memory at [mem 0x74adc000-0x74adc087]
[    0.155275] DMAR: IOMMU enabled
[    0.155276] DMAR: Intel-IOMMU: assuming all RMRRs are relaxable. This can lead to instability or data loss
[    0.372943] DMAR: Host address width 39
[    0.372944] DMAR: DRHD base: 0x000000fed90000 flags: 0x0
[    0.372951] DMAR: dmar0: reg_base_addr fed90000 ver 4:0 cap 1c0000c40660462 ecap 29a00f0505e
[    0.372953] DMAR: DRHD base: 0x000000fed91000 flags: 0x1
[    0.372957] DMAR: dmar1: reg_base_addr fed91000 ver 1:0 cap d2008c40660462 ecap f050da
[    0.372959] DMAR: RMRR base: 0x0000007c000000 end: 0x000000807fffff
[    0.372961] DMAR-IR: IOAPIC id 2 under DRHD base  0xfed91000 IOMMU 1
[    0.372962] DMAR-IR: HPET id 0 under DRHD base 0xfed91000
[    0.372963] DMAR-IR: Queued invalidation will be enabled to support x2apic and Intr-remapping.
[    0.374575] DMAR-IR: Enabled IRQ remapping in x2apic mode
[    2.847306] pci 0000:00:02.0: DMAR: Skip IOMMU disabling for graphics
[    2.913544] DMAR: No ATSR found
[    2.913544] DMAR: No SATC found
[    2.913545] DMAR: IOMMU feature fl1gp_support inconsistent
[    2.913546] DMAR: IOMMU feature pgsel_inv inconsistent
[    2.913547] DMAR: IOMMU feature nwfs inconsistent
[    2.913547] DMAR: IOMMU feature dit inconsistent
[    2.913548] DMAR: IOMMU feature sc_support inconsistent
[    2.913548] DMAR: IOMMU feature dev_iotlb_support inconsistent
[    2.913549] DMAR: dmar0: Using Queued invalidation
[    2.913551] DMAR: dmar1: Using Queued invalidation
[    2.914365] DMAR: Intel(R) Virtualization Technology for Directed I/O


dmesg | grep -i iommu
Code: 
[    0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-6.8.4-3-pve root=/dev/mapper/pve-root ro quiet intel_iommu=on,relax_rmrr iommu=pt intermap=no_x2apic_optout pcie_port_pm=off pcie_aspm.policy=performance
[    0.155229] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-6.8.4-3-pve root=/dev/mapper/pve-root ro quiet intel_iommu=on,relax_rmrr iommu=pt intermap=no_x2apic_optout pcie_port_pm=off pcie_aspm.policy=performance
[    0.155275] DMAR: IOMMU enabled
[    0.155276] DMAR: Intel-IOMMU: assuming all RMRRs are relaxable. This can lead to instability or data loss
[    0.372961] DMAR-IR: IOAPIC id 2 under DRHD base  0xfed91000 IOMMU 1
[    2.847306] pci 0000:00:02.0: DMAR: Skip IOMMU disabling for graphics
[    2.870912] iommu: Default domain type: Passthrough (set via kernel command line)
[    2.913545] DMAR: IOMMU feature fl1gp_support inconsistent
[    2.913546] DMAR: IOMMU feature pgsel_inv inconsistent
[    2.913547] DMAR: IOMMU feature nwfs inconsistent
[    2.913547] DMAR: IOMMU feature dit inconsistent
[    2.913548] DMAR: IOMMU feature sc_support inconsistent
[    2.913548] DMAR: IOMMU feature dev_iotlb_support inconsistent
[    2.913800] pci 0000:00:02.0: Adding to iommu group 0
[    2.913823] pci 0000:00:00.0: Adding to iommu group 1
[    2.913832] pci 0000:00:01.0: Adding to iommu group 2
[    2.913841] pci 0000:00:06.0: Adding to iommu group 3
[    2.913846] pci 0000:00:08.0: Adding to iommu group 4
[    2.913857] pci 0000:00:14.0: Adding to iommu group 5
[    2.913862] pci 0000:00:14.2: Adding to iommu group 5
[    2.913871] pci 0000:00:16.0: Adding to iommu group 6
[    2.913878] pci 0000:00:17.0: Adding to iommu group 7
[    2.913891] pci 0000:00:1c.0: Adding to iommu group 8
[    2.913900] pci 0000:00:1c.4: Adding to iommu group 9
[    2.913915] pci 0000:00:1f.0: Adding to iommu group 10
[    2.913923] pci 0000:00:1f.3: Adding to iommu group 10
[    2.913928] pci 0000:00:1f.4: Adding to iommu group 10
[    2.913934] pci 0000:00:1f.5: Adding to iommu group 10
[    2.913943] pci 0000:01:00.0: Adding to iommu group 11
[    2.913950] pci 0000:02:00.0: Adding to iommu group 12
[    2.913964] pci 0000:03:00.0: Adding to iommu group 13
[    2.913986] pci 0000:04:00.0: Adding to iommu group 14
[    2.914007] pci 0000:04:01.0: Adding to iommu group 15
[    2.914029] pci 0000:04:02.0: Adding to iommu group 16
[    2.914050] pci 0000:04:03.0: Adding to iommu group 17
[    2.914083] pci 0000:05:00.0: Adding to iommu group 18
[    2.914114] pci 0000:07:00.0: Adding to iommu group 19
[    2.914123] pci 0000:6e:00.0: Adding to iommu group 20

dmesg | grep -i vfio
Code: 
[    3.337610] VFIO - User Level meta-driver version: 0.3
[    3.340536] vfio_pci: add [8086:43b8[ffffffff:ffffffff]] class 0x000000/00000000
[    3.340539] vfio_pci: add [8086:1136[ffffffff:ffffffff]] class 0x000000/00000000
[    3.340542] vfio_pci: add [8086:1137[ffffffff:ffffffff]] class 0x000000/00000000
[    3.340544] vfio_pci: add [8086:1138[ffffffff:ffffffff]] class 0x000000/00000000
[ 8407.286538] vfio-pci: probe of 0000:00:1c.0 failed with error -22

ls -l /dev/vfio
Code: 
total 0
crw------- 1 root root 242,   0 Jun 15 13:37 18
crw------- 1 root root 242,   1 Jun 15 13:37 19
drwxr-xr-x 2 root root       80 Jun 15 13:37 devices
crw-rw-rw- 1 root root  10, 196 Jun 15 13:37 vfio

lspci -t
Code: 
-[0000:00]-+-00.0
           +-01.0-[01]----00.0
           +-02.0
           +-06.0-[02]----00.0
           +-08.0
           +-14.0
           +-14.2
           +-16.0
           +-17.0
           +-1c.0-[03-6d]----00.0-[04-08]--+-00.0-[05]----00.0
           |                               +-01.0-[06]--
           |                               +-02.0-[07]----00.0
           |                               \-03.0-[08]--
           +-1c.4-[6e]----00.0
           +-1f.0
           +-1f.3
           +-1f.4
           \-1f.5

/sys/bus/pci/devices/0000:00:1c.0/driver_override
Code: 
vfio-pci
[/CODE]
 
Last edited:
lspci -nnPPk
Code: 
00:00.0 Host bridge [0600]: Intel Corporation Device [8086:4c53] (rev 01)
        DeviceName: Onboard - Other
        Subsystem: Micro-Star International Co., Ltd. [MSI] Device [1462:7d04]
        Kernel driver in use: icl_uncore
00:01.0 PCI bridge [0604]: Intel Corporation Device [8086:4c01] (rev 01)
        Subsystem: Micro-Star International Co., Ltd. [MSI] Device [1462:7d04]
        Kernel driver in use: pcieport
00:02.0 VGA compatible controller [0300]: Intel Corporation RocketLake-S GT1 [UHD Graphics 750] [8086:4c8a] (rev 04)
        DeviceName: Onboard - Video
        Subsystem: Micro-Star International Co., Ltd. [MSI] RocketLake-S GT1 [UHD Graphics 750] [1462:7d04]
        Kernel driver in use: i915
        Kernel modules: i915, xe
00:06.0 PCI bridge [0604]: Intel Corporation Device [8086:4c09] (rev 01)
        Kernel driver in use: pcieport
00:08.0 System peripheral [0880]: Intel Corporation Device [8086:4c11] (rev 01)
        DeviceName: Onboard - Other
        Subsystem: Micro-Star International Co., Ltd. [MSI] Device [1462:7d04]
00:14.0 USB controller [0c03]: Intel Corporation Tiger Lake-H USB 3.2 Gen 2x1 xHCI Host Controller [8086:43ed] (rev 11)
        DeviceName: Onboard - Other
        Subsystem: Micro-Star International Co., Ltd. [MSI] Tiger Lake-H USB 3.2 Gen 2x1 xHCI Host Controller [1462:7d04]
        Kernel driver in use: xhci_hcd
        Kernel modules: xhci_pci
00:14.2 RAM memory [0500]: Intel Corporation Tiger Lake-H Shared SRAM [8086:43ef] (rev 11)
        DeviceName: Onboard - Other
00:16.0 Communication controller [0780]: Intel Corporation Tiger Lake-H Management Engine Interface [8086:43e0] (rev 11)
        DeviceName: Onboard - Other
        Subsystem: Micro-Star International Co., Ltd. [MSI] Tiger Lake-H Management Engine Interface [1462:7d04]
        Kernel driver in use: mei_me
        Kernel modules: mei_me
00:17.0 SATA controller [0106]: Intel Corporation Device [8086:43d2] (rev 11)
        DeviceName: Onboard - SATA
        Subsystem: Micro-Star International Co., Ltd. [MSI] Device [1462:7d04]
        Kernel driver in use: ahci
        Kernel modules: ahci
00:1c.0 PCI bridge [0604]: Intel Corporation Tiger Lake-H PCIe Root Port [8086:43b8] (rev 11)
        Subsystem: Micro-Star International Co., Ltd. [MSI] Tiger Lake-H PCIe Root Port [1462:7d04]
        Kernel driver in use: pcieport
00:1c.4 PCI bridge [0604]: Intel Corporation Tiger Lake-H PCI Express Root Port #5 [8086:43bc] (rev 11)
        Subsystem: Micro-Star International Co., Ltd. [MSI] Tiger Lake-H PCI Express Root Port [1462:7d04]
        Kernel driver in use: pcieport
00:1f.0 ISA bridge [0601]: Intel Corporation Z590 LPC/eSPI Controller [8086:4385] (rev 11)
        DeviceName: Onboard - Other
        Subsystem: Micro-Star International Co., Ltd. [MSI] Z590 LPC/eSPI Controller [1462:7d04]
00:1f.3 Audio device [0403]: Intel Corporation Tiger Lake-H HD Audio Controller [8086:43c8] (rev 11)
        DeviceName: Onboard - Sound
        Subsystem: Micro-Star International Co., Ltd. [MSI] Tiger Lake-H HD Audio Controller [1462:7d04]
        Kernel driver in use: snd_hda_intel
        Kernel modules: snd_hda_intel, snd_sof_pci_intel_tgl
00:1f.4 SMBus [0c05]: Intel Corporation Tiger Lake-H SMBus Controller [8086:43a3] (rev 11)
        DeviceName: Onboard - Other
        Subsystem: Micro-Star International Co., Ltd. [MSI] Tiger Lake-H SMBus Controller [1462:7d04]
        Kernel driver in use: i801_smbus
        Kernel modules: i2c_i801
00:1f.5 Serial bus controller [0c80]: Intel Corporation Tiger Lake-H SPI Controller [8086:43a4] (rev 11)
        DeviceName: Onboard - Other
        Subsystem: Micro-Star International Co., Ltd. [MSI] Tiger Lake-H SPI Controller [1462:7d04]
        Kernel driver in use: intel-spi
        Kernel modules: spi_intel_pci
00:01.0/01:00.0 Mass storage controller [0180]: Device [ace1:0018] (rev 01)
        Subsystem: Device [ace1:0018]
00:06.0/02:00.0 Non-Volatile memory controller [0108]: Seagate Technology PLC FireCuda 530 SSD [1bb1:5018] (rev 01)
        Subsystem: Seagate Technology PLC FireCuda 530 SSD [1bb1:5018]
        Kernel driver in use: nvme
        Kernel modules: nvme
00:1c.0/03:00.0 PCI bridge [0604]: Intel Corporation Thunderbolt 4 Bridge [Maple Ridge 4C 2020] [8086:1136] (rev 02)
        Subsystem: Intel Corporation Thunderbolt 4 Bridge [Maple Ridge 4C 2020] [8086:0000]
        Kernel driver in use: pcieport
00:1c.0/03:00.0/04:00.0 PCI bridge [0604]: Intel Corporation Thunderbolt 4 Bridge [Maple Ridge 4C 2020] [8086:1136] (rev 02)
        Subsystem: Intel Corporation Thunderbolt 4 Bridge [Maple Ridge 4C 2020] [8086:0000]
        Kernel driver in use: pcieport
00:1c.0/03:00.0/04:01.0 PCI bridge [0604]: Intel Corporation Thunderbolt 4 Bridge [Maple Ridge 4C 2020] [8086:1136] (rev 02)
        Subsystem: Intel Corporation Thunderbolt 4 Bridge [Maple Ridge 4C 2020] [8086:0000]
        Kernel driver in use: pcieport
00:1c.0/03:00.0/04:02.0 PCI bridge [0604]: Intel Corporation Thunderbolt 4 Bridge [Maple Ridge 4C 2020] [8086:1136] (rev 02)
        Subsystem: Intel Corporation Thunderbolt 4 Bridge [Maple Ridge 4C 2020] [8086:0000]
        Kernel driver in use: pcieport
00:1c.0/03:00.0/04:03.0 PCI bridge [0604]: Intel Corporation Thunderbolt 4 Bridge [Maple Ridge 4C 2020] [8086:1136] (rev 02)
        Subsystem: Intel Corporation Thunderbolt 4 Bridge [Maple Ridge 4C 2020] [8086:0000]
        Kernel driver in use: pcieport
00:1c.0/03:00.0/04:00.0/05:00.0 USB controller [0c03]: Intel Corporation Thunderbolt 4 NHI [Maple Ridge 4C 2020] [8086:1137]
        Subsystem: Intel Corporation Thunderbolt 4 NHI [Maple Ridge 4C 2020] [8086:0000]
        Kernel driver in use: vfio-pci
        Kernel modules: thunderbolt
00:1c.0/03:00.0/04:02.0/07:00.0 USB controller [0c03]: Intel Corporation Thunderbolt 4 USB Controller [Maple Ridge 4C 2020] [8086:1138]
        Kernel driver in use: vfio-pci
        Kernel modules: xhci_pci
00:1c.4/6e:00.0 Ethernet controller [0200]: Intel Corporation Ethernet Controller I225-V [8086:15f3] (rev 03)
        Subsystem: Micro-Star International Co., Ltd. [MSI] Ethernet Controller I225-V [1462:7d04]
        Kernel driver in use: igc
        Kernel modules: igc

for d in /sys/kernel/iommu_groups/*/devices/*; do echo "$d"; lspci -nns "${d##*/}"; done | less
Code: 
/sys/kernel/iommu_groups/0/devices/0000:00:02.0
00:02.0 VGA compatible controller [0300]: Intel Corporation RocketLake-S GT1 [UHD Graphics 750] [8086:4c8a] (rev 04)
/sys/kernel/iommu_groups/10/devices/0000:00:1f.0
00:1f.0 ISA bridge [0601]: Intel Corporation Z590 LPC/eSPI Controller [8086:4385] (rev 11)
/sys/kernel/iommu_groups/10/devices/0000:00:1f.3
00:1f.3 Audio device [0403]: Intel Corporation Tiger Lake-H HD Audio Controller [8086:43c8] (rev 11)
/sys/kernel/iommu_groups/10/devices/0000:00:1f.4
00:1f.4 SMBus [0c05]: Intel Corporation Tiger Lake-H SMBus Controller [8086:43a3] (rev 11)
/sys/kernel/iommu_groups/10/devices/0000:00:1f.5
00:1f.5 Serial bus controller [0c80]: Intel Corporation Tiger Lake-H SPI Controller [8086:43a4] (rev 11)
/sys/kernel/iommu_groups/11/devices/0000:01:00.0
01:00.0 Mass storage controller [0180]: Device [ace1:0018] (rev 01)
/sys/kernel/iommu_groups/12/devices/0000:02:00.0
02:00.0 Non-Volatile memory controller [0108]: Seagate Technology PLC FireCuda 530 SSD [1bb1:5018] (rev 01)
/sys/kernel/iommu_groups/13/devices/0000:03:00.0
03:00.0 PCI bridge [0604]: Intel Corporation Thunderbolt 4 Bridge [Maple Ridge 4C 2020] [8086:1136] (rev 02)
/sys/kernel/iommu_groups/14/devices/0000:04:00.0
04:00.0 PCI bridge [0604]: Intel Corporation Thunderbolt 4 Bridge [Maple Ridge 4C 2020] [8086:1136] (rev 02)
/sys/kernel/iommu_groups/15/devices/0000:04:01.0
04:01.0 PCI bridge [0604]: Intel Corporation Thunderbolt 4 Bridge [Maple Ridge 4C 2020] [8086:1136] (rev 02)
/sys/kernel/iommu_groups/16/devices/0000:04:02.0
04:02.0 PCI bridge [0604]: Intel Corporation Thunderbolt 4 Bridge [Maple Ridge 4C 2020] [8086:1136] (rev 02)
/sys/kernel/iommu_groups/17/devices/0000:04:03.0
04:03.0 PCI bridge [0604]: Intel Corporation Thunderbolt 4 Bridge [Maple Ridge 4C 2020] [8086:1136] (rev 02)
/sys/kernel/iommu_groups/18/devices/0000:05:00.0
05:00.0 USB controller [0c03]: Intel Corporation Thunderbolt 4 NHI [Maple Ridge 4C 2020] [8086:1137]
/sys/kernel/iommu_groups/19/devices/0000:07:00.0
07:00.0 USB controller [0c03]: Intel Corporation Thunderbolt 4 USB Controller [Maple Ridge 4C 2020] [8086:1138]
/sys/kernel/iommu_groups/1/devices/0000:00:00.0
00:00.0 Host bridge [0600]: Intel Corporation Device [8086:4c53] (rev 01)
/sys/kernel/iommu_groups/20/devices/0000:6e:00.0
6e:00.0 Ethernet controller [0200]: Intel Corporation Ethernet Controller I225-V [8086:15f3] (rev 03)
/sys/kernel/iommu_groups/2/devices/0000:00:01.0
00:01.0 PCI bridge [0604]: Intel Corporation Device [8086:4c01] (rev 01)
/sys/kernel/iommu_groups/3/devices/0000:00:06.0
00:06.0 PCI bridge [0604]: Intel Corporation Device [8086:4c09] (rev 01)
/sys/kernel/iommu_groups/4/devices/0000:00:08.0
00:08.0 System peripheral [0880]: Intel Corporation Device [8086:4c11] (rev 01)
/sys/kernel/iommu_groups/5/devices/0000:00:14.0
00:14.0 USB controller [0c03]: Intel Corporation Tiger Lake-H USB 3.2 Gen 2x1 xHCI Host Controller [8086:43ed] (rev 11)
/sys/kernel/iommu_groups/5/devices/0000:00:14.2
00:14.2 RAM memory [0500]: Intel Corporation Tiger Lake-H Shared SRAM [8086:43ef] (rev 11)
/sys/kernel/iommu_groups/6/devices/0000:00:16.0
00:16.0 Communication controller [0780]: Intel Corporation Tiger Lake-H Management Engine Interface [8086:43e0] (rev 11)
/sys/kernel/iommu_groups/7/devices/0000:00:17.0
00:17.0 SATA controller [0106]: Intel Corporation Device [8086:43d2] (rev 11)
/sys/kernel/iommu_groups/8/devices/0000:00:1c.0
00:1c.0 PCI bridge [0604]: Intel Corporation Tiger Lake-H PCIe Root Port [8086:43b8] (rev 11)
/sys/kernel/iommu_groups/9/devices/0000:00:1c.4
00:1c.4 PCI bridge [0604]: Intel Corporation Tiger Lake-H PCI Express Root Port #5 [8086:43bc] (rev 11)

find /sys/kernel/iommu_groups/ -type l
Code: 
/sys/kernel/iommu_groups/17/devices/0000:04:03.0
/sys/kernel/iommu_groups/7/devices/0000:00:17.0
/sys/kernel/iommu_groups/15/devices/0000:04:01.0
/sys/kernel/iommu_groups/5/devices/0000:00:14.2
/sys/kernel/iommu_groups/5/devices/0000:00:14.0
/sys/kernel/iommu_groups/13/devices/0000:03:00.0
/sys/kernel/iommu_groups/3/devices/0000:00:06.0
/sys/kernel/iommu_groups/11/devices/0000:01:00.0
/sys/kernel/iommu_groups/1/devices/0000:00:00.0
/sys/kernel/iommu_groups/18/devices/0000:05:00.0
/sys/kernel/iommu_groups/8/devices/0000:00:1c.0
/sys/kernel/iommu_groups/16/devices/0000:04:02.0
/sys/kernel/iommu_groups/6/devices/0000:00:16.0
/sys/kernel/iommu_groups/14/devices/0000:04:00.0
/sys/kernel/iommu_groups/4/devices/0000:00:08.0
/sys/kernel/iommu_groups/12/devices/0000:02:00.0
/sys/kernel/iommu_groups/2/devices/0000:00:01.0
/sys/kernel/iommu_groups/20/devices/0000:6e:00.0
/sys/kernel/iommu_groups/10/devices/0000:00:1f.0
/sys/kernel/iommu_groups/10/devices/0000:00:1f.5
/sys/kernel/iommu_groups/10/devices/0000:00:1f.3
/sys/kernel/iommu_groups/10/devices/0000:00:1f.4
/sys/kernel/iommu_groups/0/devices/0000:00:02.0
/sys/kernel/iommu_groups/19/devices/0000:07:00.0
/sys/kernel/iommu_groups/9/devices/0000:00:1c.4

so in the end the pcie and the thunderbolt are
Code: 
pcie 0000:00:1c.0
thunderbolt bridge 0000:03:00.0
thunderbolt bridge 0000:04:00.0
thunderbolt bridge 0000:04:01.0
thunderbolt bridge 0000:04:02.0
thunderbolt bridge 0000:04:03.0
thunderbolt nhi 0000:05:00.0
thunderbolt usb 0000:07:00.0



and this is the vm config a windows 10 /etc/pve/qemu-server/100.conf
Code: 
#args%3A -device qemu-xhci,id=xhci1 -device usb-host,bus=xhci1.0,hostbus=2,hostport=1,id=Device_1 -device usb-host,bus=xhci1.0,hostbus=1,hostport=1.7,id=Device_2
agent: 1
args: -cpu 'host,-hypervisor,kvm=off'
audio0: device=ich9-intel-hda,driver=none
bios: ovmf
boot: order=scsi0;sata0
cores: 2
cpu: host,flags=+hv-evmcs
efidisk0: local-lvm:vm-100-disk-0,efitype=4m,pre-enrolled-keys=1,size=4M
hostpci0: 0000:00:1c.0,pcie=1
hostpci1: 0000:03:00.0,pcie=1
hostpci2: 0000:04:00.0,pcie=1
hostpci3: 0000:04:01.0,pcie=1
hostpci4: 0000:04:02.0,pcie=1
hostpci5: 0000:04:03.0,pcie=1
hostpci6: 0000:05:00.0,pcie=1
hostpci7: 0000:07:00.0,pcie=1
hotplug: usb
machine: pc-q35-8.1
memory: 8192
meta: creation-qemu=8.1.2,ctime=1701594220
name: win10base
net0: virtio=BC:24:11:CB:47:85,bridge=vmbr0
numa: 0
ostype: win10
sata0: none,media=cdrom
scsi0: local-lvm:vm-100-disk-1,cache=writeback,discard=on,size=64G,ssd=1
scsihw: virtio-scsi-pci
smbios1: uuid=85095253-388e-46d7-a52e-c5ac1df333f3
sockets: 2
vga: std,memory=256
vmgenid: dd77a721-0440-47a8-a25a-5608676959ac
it giveme error when i try to run it
Code: 
kvm: -device vfio-pci,host=0000:00:1c.0,id=hostpci0,bus=ich9-pcie-port-1,addr=0x0: vfio 0000:00:1c.0: failed to open /dev/vfio/8: No such file or directory
TASK ERROR: start failed: QEMU exited with code 1

if i map the 0000:00:1c it maps 0000:00:1c.0 and 0000:00:1c.4 Tyger Lake-H pci express root port #5
happend the same

i try
Code: 
echo 0000:00:1c.0 | sudo tee /sys/bus/pci/devices/0000:00:1c.0/driver/unbind
0000:00:1c.0
and then
Code: 
echo 8086 43b8 | sudo tee /sys/bus/pci/drivers/vfio-pci/new_id
tee: /sys/bus/pci/drivers/vfio-pci/new_id: File exists
echo 0000:00:1c.0 | sudo tee /sys/bus/pci/drivers/vfio-pci/bind
tee: /sys/bus/pci/drivers/vfio-pci/bind: Invalid argument


so im out of ideas if anyone could helpme
 
also looking for this. i wasn't able to get this working. i'm trying to run a daily driver ubuntu QEMU in proxmox but I wanted to get single cable so everything is going through single thunderbolt connection. (7900 xtx going in to Asus ProArt z790 DP ins)
 
  • Like
Reactions: mikeone
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom