Pcie/Thunderbolt 4/USB4 Passthrough on Proxmox VE 8.2

S

ss4sgoku

New Member
Jan 17, 2024
5
0
1
hi, this is my second post im a newbie in proxmox so im trying to figure out how to acomplish a full passthrough of my thunderbolt4 that lookslike its conencted via pcie port that port share conection with the network card
First, the system has a Motherboard MSI Z590 ACE with a i5-11600k it has a thunderbolt 4 with intel jhl8540 via pcie slot, Proxmox 8.2 full updated

This is the diagram of the Motherboard z590 ace
and this is the thunderbolt module i have this pictures from this web


This is my grub right now
Code: 
GRUB_DEFAULT=0
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian`
GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on,relax_rmrr iommu=pt intermap=no_x2ap  pcie_port_pm=off pcie_aspm.policy=performance"
GRUB_CMDLINE_LINUX=""

cat /etc/modules
Code: 
vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd
thunderbolt

dmesg | grep 'remapping'
Code: 
[    0.372963] DMAR-IR: Queued invalidation will be enabled to support x2apic and Intr-remapping.
[    0.374575] DMAR-IR: Enabled IRQ remapping in x2apic mode

dmesg | grep 'Intel-IOMMU'
Code: 
[    0.155276] DMAR: Intel-IOMMU: assuming all RMRRs are relaxable. This can lead to instability or data loss
ç

cat /proc/modules | grep pci
Code: 
snd_sof_pci_intel_tgl 12288 0 - Live 0xffffffffc22a3000
snd_sof_intel_hda_common 208896 1 snd_sof_pci_intel_tgl, Live 0xffffffffc225c000
snd_sof_pci 24576 2 snd_sof_pci_intel_tgl,snd_sof_intel_hda_common, Live 0xffffffffc21fb000
snd_sof 356352 3 snd_sof_intel_hda_common,snd_sof_intel_hda,snd_sof_pci, Live 0xffffffffc217f000
snd_soc_acpi_intel_match 102400 2 snd_sof_pci_intel_tgl,snd_sof_intel_hda_common, Live 0xffffffffc2132000
xhci_pci 24576 0 - Live 0xffffffffc04c0000
xhci_pci_renesas 16384 1 xhci_pci, Live 0xffffffffc046a000
spi_intel_pci 12288 0 - Live 0xffffffffc066f000
xhci_hcd 364544 1 xhci_pci, Live 0xffffffffc0599000
spi_intel 32768 1 spi_intel_pci, Live 0xffffffffc0528000
vfio_pci 16384 0 - Live 0xffffffffc049b000
vfio_pci_core 86016 1 vfio_pci, Live 0xffffffffc0478000
irqbypass 12288 2 kvm,vfio_pci_core, Live 0xffffffffc0470000
vfio 69632 3 vfio_pci,vfio_pci_core,vfio_iommu_type1, Live 0xffffffffc043a000

cat /proc/modules | grep usb
Code: 
snd_usb_audio 442368 1 - Live 0xffffffffc1d43000
snd_usbmidi_lib 49152 1 snd_usb_audio, Live 0xffffffffc1d0b000
snd_ump 40960 1 snd_usb_audio, Live 0xffffffffc1cef000
snd_rawmidi 53248 2 snd_usbmidi_lib,snd_ump, Live 0xffffffffc1c76000
mc 86016 1 snd_usb_audio, Live 0xffffffffc1acb000
snd_hwdep 20480 2 snd_usb_audio,snd_hda_codec, Live 0xffffffffc1a8e000
snd_pcm 188416 12 snd_hda_codec_hdmi,snd_sof_intel_hda_common,soundwire_intel,snd_sof,snd_sof_utils,snd_soc_core,snd_compress,snd_pcm_dmaengine,snd_hda_intel,snd_usb_audio,snd_hda_codec,snd_hda_core, Live 0xffffffffc1b4c000
snd 139264 21 snd_seq,snd_hda_codec_hdmi,snd_sof,snd_soc_core,snd_compress,snd_hda_intel,snd_usb_audio,snd_usbmidi_lib,snd_ump,snd_hda_codec,snd_rawmidi,snd_seq_device,snd_hwdep,snd_pcm,snd_timer, Live 0xffffffffc1a9e000
usbmouse 12288 0 - Live 0xffffffffc0b04000
usbhid 69632 0 - Live 0xffffffffc0aed000
hid 167936 2 hid_generic,usbhid, Live 0xffffffffc0ab8000

pvesh get /nodes/pve/hardware/pci --pci-class-blacklist ""
Code: 
│ class    │ device │ id           │ iommugroup │ vendor │ device_name                                        │ mdev │ subsystem_device │ subsystem_device_name │ subsystem_vendor │ subsystem_vendor_name       
╞══════════╪════════╪══════════════╪════════════╪════════╪════════════════════════════════════════════════════╪══════╪══════════════════╪═══════════════════════╪══════════════════╪═════════════════════════════
│ 0x010601 │ 0x43d2 │ 0000:00:17.0 │          7 │ 0x8086 │                                                    │      │ 0x7d04           │                       │ 0x1462           │ Micro-Star International Co.
├──────────┼────────┼──────────────┼────────────┼────────┼────────────────────────────────────────────────────┼──────┼──────────────────┼───────────────────────┼──────────────────┼─────────────────────────────
│ 0x010802 │ 0x5018 │ 0000:02:00.0 │         12 │ 0x1bb1 │ FireCuda 530 SSD                                   │      │ 0x5018           │                       │ 0x1bb1           │ Seagate Technology PLC     
├──────────┼────────┼──────────────┼────────────┼────────┼────────────────────────────────────────────────────┼──────┼──────────────────┼───────────────────────┼──────────────────┼─────────────────────────────
│ 0x018000 │ 0x0018 │ 0000:01:00.0 │         11 │ 0xace1 │                                                    │      │ 0x0018           │                       │ 0xace1           │                             
├──────────┼────────┼──────────────┼────────────┼────────┼────────────────────────────────────────────────────┼──────┼──────────────────┼───────────────────────┼──────────────────┼─────────────────────────────
│ 0x020000 │ 0x15f3 │ 0000:6e:00.0 │         20 │ 0x8086 │ Ethernet Controller I225-V                         │      │ 0x7d04           │                       │ 0x1462           │ Micro-Star International Co.
├──────────┼────────┼──────────────┼────────────┼────────┼────────────────────────────────────────────────────┼──────┼──────────────────┼───────────────────────┼──────────────────┼─────────────────────────────
│ 0x030000 │ 0x4c8a │ 0000:00:02.0 │          0 │ 0x8086 │ RocketLake-S GT1 [UHD Graphics 750]                │      │ 0x7d04           │                       │ 0x1462           │ Micro-Star International Co.
├──────────┼────────┼──────────────┼────────────┼────────┼────────────────────────────────────────────────────┼──────┼──────────────────┼───────────────────────┼──────────────────┼─────────────────────────────
│ 0x040300 │ 0x43c8 │ 0000:00:1f.3 │         10 │ 0x8086 │ Tiger Lake-H HD Audio Controller                   │      │ 0x7d04           │                       │ 0x1462           │ Micro-Star International Co.
├──────────┼────────┼──────────────┼────────────┼────────┼────────────────────────────────────────────────────┼──────┼──────────────────┼───────────────────────┼──────────────────┼─────────────────────────────
│ 0x050000 │ 0x43ef │ 0000:00:14.2 │          5 │ 0x8086 │ Tiger Lake-H Shared SRAM                           │      │ 0x0000           │                       │ 0x0000           │                             
├──────────┼────────┼──────────────┼────────────┼────────┼────────────────────────────────────────────────────┼──────┼──────────────────┼───────────────────────┼──────────────────┼─────────────────────────────
│ 0x060000 │ 0x4c53 │ 0000:00:00.0 │          1 │ 0x8086 │                                                    │      │ 0x7d04           │                       │ 0x1462           │ Micro-Star International Co.
├──────────┼────────┼──────────────┼────────────┼────────┼────────────────────────────────────────────────────┼──────┼──────────────────┼───────────────────────┼──────────────────┼─────────────────────────────
│ 0x060100 │ 0x4385 │ 0000:00:1f.0 │         10 │ 0x8086 │                                                    │      │ 0x7d04           │                       │ 0x1462           │ Micro-Star International Co.
├──────────┼────────┼──────────────┼────────────┼────────┼────────────────────────────────────────────────────┼──────┼──────────────────┼───────────────────────┼──────────────────┼─────────────────────────────
│ 0x060400 │ 0x4c01 │ 0000:00:01.0 │          2 │ 0x8086 │                                                    │      │ 0x7d04           │                       │ 0x1462           │ Micro-Star International Co.
├──────────┼────────┼──────────────┼────────────┼────────┼────────────────────────────────────────────────────┼──────┼──────────────────┼───────────────────────┼──────────────────┼─────────────────────────────
│ 0x060400 │ 0x4c09 │ 0000:00:06.0 │          3 │ 0x8086 │                                                    │      │ 0x0000           │                       │ 0x0000           │                             
├──────────┼────────┼──────────────┼────────────┼────────┼────────────────────────────────────────────────────┼──────┼──────────────────┼───────────────────────┼──────────────────┼─────────────────────────────
│ 0x060400 │ 0x43b8 │ 0000:00:1c.0 │          8 │ 0x8086 │                                                    │      │ 0x7d04           │                       │ 0x1462           │ Micro-Star International Co.
├──────────┼────────┼──────────────┼────────────┼────────┼────────────────────────────────────────────────────┼──────┼──────────────────┼───────────────────────┼──────────────────┼─────────────────────────────
│ 0x060400 │ 0x43bc │ 0000:00:1c.4 │          9 │ 0x8086 │ Tiger Lake-H PCI Express Root Port #5              │      │ 0x7d04           │                       │ 0x1462           │ Micro-Star International Co.
├──────────┼────────┼──────────────┼────────────┼────────┼────────────────────────────────────────────────────┼──────┼──────────────────┼───────────────────────┼──────────────────┼─────────────────────────────
│ 0x060400 │ 0x1136 │ 0000:03:00.0 │         13 │ 0x8086 │ Thunderbolt 4 Bridge [Maple Ridge 4C 2020]         │      │ 0x0000           │                       │ 0x8086           │ Intel Corporation           
├──────────┼────────┼──────────────┼────────────┼────────┼────────────────────────────────────────────────────┼──────┼──────────────────┼───────────────────────┼──────────────────┼─────────────────────────────
│ 0x060400 │ 0x1136 │ 0000:04:00.0 │         14 │ 0x8086 │ Thunderbolt 4 Bridge [Maple Ridge 4C 2020]         │      │ 0x0000           │                       │ 0x8086           │ Intel Corporation           
├──────────┼────────┼──────────────┼────────────┼────────┼────────────────────────────────────────────────────┼──────┼──────────────────┼───────────────────────┼──────────────────┼─────────────────────────────
│ 0x060400 │ 0x1136 │ 0000:04:01.0 │         15 │ 0x8086 │ Thunderbolt 4 Bridge [Maple Ridge 4C 2020]         │      │ 0x0000           │                       │ 0x8086           │ Intel Corporation           
├──────────┼────────┼──────────────┼────────────┼────────┼────────────────────────────────────────────────────┼──────┼──────────────────┼───────────────────────┼──────────────────┼─────────────────────────────
│ 0x060400 │ 0x1136 │ 0000:04:02.0 │         16 │ 0x8086 │ Thunderbolt 4 Bridge [Maple Ridge 4C 2020]         │      │ 0x0000           │                       │ 0x8086           │ Intel Corporation           
├──────────┼────────┼──────────────┼────────────┼────────┼────────────────────────────────────────────────────┼──────┼──────────────────┼───────────────────────┼──────────────────┼─────────────────────────────
│ 0x060400 │ 0x1136 │ 0000:04:03.0 │         17 │ 0x8086 │ Thunderbolt 4 Bridge [Maple Ridge 4C 2020]         │      │ 0x0000           │                       │ 0x8086           │ Intel Corporation           
├──────────┼────────┼──────────────┼────────────┼────────┼────────────────────────────────────────────────────┼──────┼──────────────────┼───────────────────────┼──────────────────┼─────────────────────────────
│ 0x078000 │ 0x43e0 │ 0000:00:16.0 │          6 │ 0x8086 │ Tiger Lake-H Management Engine Interface           │      │ 0x7d04           │                       │ 0x1462           │ Micro-Star International Co.
├──────────┼────────┼──────────────┼────────────┼────────┼────────────────────────────────────────────────────┼──────┼──────────────────┼───────────────────────┼──────────────────┼─────────────────────────────
│ 0x088000 │ 0x4c11 │ 0000:00:08.0 │          4 │ 0x8086 │                                                    │      │ 0x7d04           │                       │ 0x1462           │ Micro-Star International Co.
├──────────┼────────┼──────────────┼────────────┼────────┼────────────────────────────────────────────────────┼──────┼──────────────────┼───────────────────────┼──────────────────┼─────────────────────────────
│ 0x0c0330 │ 0x43ed │ 0000:00:14.0 │          5 │ 0x8086 │ Tiger Lake-H USB 3.2 Gen 2x1 xHCI Host Controller  │      │ 0x7d04           │                       │ 0x1462           │ Micro-Star International Co.
├──────────┼────────┼──────────────┼────────────┼────────┼────────────────────────────────────────────────────┼──────┼──────────────────┼───────────────────────┼──────────────────┼─────────────────────────────
│ 0x0c0330 │ 0x1138 │ 0000:07:00.0 │         19 │ 0x8086 │ Thunderbolt 4 USB Controller [Maple Ridge 4C 2020] │      │ 0x0000           │                       │ 0x0000           │                             
├──────────┼────────┼──────────────┼────────────┼────────┼────────────────────────────────────────────────────┼──────┼──────────────────┼───────────────────────┼──────────────────┼─────────────────────────────
│ 0x0c0340 │ 0x1137 │ 0000:05:00.0 │         18 │ 0x8086 │ Thunderbolt 4 NHI [Maple Ridge 4C 2020]            │      │ 0x0000           │                       │ 0x8086           │ Intel Corporation           
├──────────┼────────┼──────────────┼────────────┼────────┼────────────────────────────────────────────────────┼──────┼──────────────────┼───────────────────────┼──────────────────┼─────────────────────────────
│ 0x0c0500 │ 0x43a3 │ 0000:00:1f.4 │         10 │ 0x8086 │ Tiger Lake-H SMBus Controller                      │      │ 0x7d04           │                       │ 0x1462           │ Micro-Star International Co.
├──────────┼────────┼──────────────┼────────────┼────────┼────────────────────────────────────────────────────┼──────┼──────────────────┼───────────────────────┼──────────────────┼─────────────────────────────
│ 0x0c8000 │ 0x43a4 │ 0000:00:1f.5 │         10 │ 0x8086 │ Tiger Lake-H SPI Controller                        │      │ 0x7d04           │                       │ 0x1462           │ Micro-Star International Co.
└──────────┴────────┴──────────────┴────────────┴────────┴────────────────────────────────────────────────────┴──────┴──────────────────┴───────────────────────┴──────────────────┴─────────────────────────────
 
/etc/modprobe.d/vfio.conf
Code: 
options vfio-pci ids=8086:43b8,8086:1136,8086:1137,8086:1138
softdep thunderbolt pre: vfio-pci
softdep pcieport pre: vfio-pci
softdep xhci_pci pre: vfio-pci
softdep xhci_hcd pre: vfio-pci

/etc/modprobe.d/blacklist.conf
Code: 
blacklist pcieport
blacklist xhci_hcd
blacklist thunderbolt

/etc/modprobe.d/kvm.conf
Code: 
options kvm ignore_msrs=1

/etc/modprobe.d/iommu_unsafe_interrupts.conf
Code: 
options vfio_iommu_type1 allow_unsafe_interrupts=1

dmesg | grep -e DMAR -e IOMMU
Code: 
[    0.016187] ACPI: DMAR 0x0000000074ADC000 000088 (v02 INTEL  EDK2     00000002      01000013)
[    0.016215] ACPI: Reserving DMAR table memory at [mem 0x74adc000-0x74adc087]
[    0.155275] DMAR: IOMMU enabled
[    0.155276] DMAR: Intel-IOMMU: assuming all RMRRs are relaxable. This can lead to instability or data loss
[    0.372943] DMAR: Host address width 39
[    0.372944] DMAR: DRHD base: 0x000000fed90000 flags: 0x0
[    0.372951] DMAR: dmar0: reg_base_addr fed90000 ver 4:0 cap 1c0000c40660462 ecap 29a00f0505e
[    0.372953] DMAR: DRHD base: 0x000000fed91000 flags: 0x1
[    0.372957] DMAR: dmar1: reg_base_addr fed91000 ver 1:0 cap d2008c40660462 ecap f050da
[    0.372959] DMAR: RMRR base: 0x0000007c000000 end: 0x000000807fffff
[    0.372961] DMAR-IR: IOAPIC id 2 under DRHD base  0xfed91000 IOMMU 1
[    0.372962] DMAR-IR: HPET id 0 under DRHD base 0xfed91000
[    0.372963] DMAR-IR: Queued invalidation will be enabled to support x2apic and Intr-remapping.
[    0.374575] DMAR-IR: Enabled IRQ remapping in x2apic mode
[    2.847306] pci 0000:00:02.0: DMAR: Skip IOMMU disabling for graphics
[    2.913544] DMAR: No ATSR found
[    2.913544] DMAR: No SATC found
[    2.913545] DMAR: IOMMU feature fl1gp_support inconsistent
[    2.913546] DMAR: IOMMU feature pgsel_inv inconsistent
[    2.913547] DMAR: IOMMU feature nwfs inconsistent
[    2.913547] DMAR: IOMMU feature dit inconsistent
[    2.913548] DMAR: IOMMU feature sc_support inconsistent
[    2.913548] DMAR: IOMMU feature dev_iotlb_support inconsistent
[    2.913549] DMAR: dmar0: Using Queued invalidation
[    2.913551] DMAR: dmar1: Using Queued invalidation
[    2.914365] DMAR: Intel(R) Virtualization Technology for Directed I/O


dmesg | grep -i iommu
Code: 
[    0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-6.8.4-3-pve root=/dev/mapper/pve-root ro quiet intel_iommu=on,relax_rmrr iommu=pt intermap=no_x2apic_optout pcie_port_pm=off pcie_aspm.policy=performance
[    0.155229] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-6.8.4-3-pve root=/dev/mapper/pve-root ro quiet intel_iommu=on,relax_rmrr iommu=pt intermap=no_x2apic_optout pcie_port_pm=off pcie_aspm.policy=performance
[    0.155275] DMAR: IOMMU enabled
[    0.155276] DMAR: Intel-IOMMU: assuming all RMRRs are relaxable. This can lead to instability or data loss
[    0.372961] DMAR-IR: IOAPIC id 2 under DRHD base  0xfed91000 IOMMU 1
[    2.847306] pci 0000:00:02.0: DMAR: Skip IOMMU disabling for graphics
[    2.870912] iommu: Default domain type: Passthrough (set via kernel command line)
[    2.913545] DMAR: IOMMU feature fl1gp_support inconsistent
[    2.913546] DMAR: IOMMU feature pgsel_inv inconsistent
[    2.913547] DMAR: IOMMU feature nwfs inconsistent
[    2.913547] DMAR: IOMMU feature dit inconsistent
[    2.913548] DMAR: IOMMU feature sc_support inconsistent
[    2.913548] DMAR: IOMMU feature dev_iotlb_support inconsistent
[    2.913800] pci 0000:00:02.0: Adding to iommu group 0
[    2.913823] pci 0000:00:00.0: Adding to iommu group 1
[    2.913832] pci 0000:00:01.0: Adding to iommu group 2
[    2.913841] pci 0000:00:06.0: Adding to iommu group 3
[    2.913846] pci 0000:00:08.0: Adding to iommu group 4
[    2.913857] pci 0000:00:14.0: Adding to iommu group 5
[    2.913862] pci 0000:00:14.2: Adding to iommu group 5
[    2.913871] pci 0000:00:16.0: Adding to iommu group 6
[    2.913878] pci 0000:00:17.0: Adding to iommu group 7
[    2.913891] pci 0000:00:1c.0: Adding to iommu group 8
[    2.913900] pci 0000:00:1c.4: Adding to iommu group 9
[    2.913915] pci 0000:00:1f.0: Adding to iommu group 10
[    2.913923] pci 0000:00:1f.3: Adding to iommu group 10
[    2.913928] pci 0000:00:1f.4: Adding to iommu group 10
[    2.913934] pci 0000:00:1f.5: Adding to iommu group 10
[    2.913943] pci 0000:01:00.0: Adding to iommu group 11
[    2.913950] pci 0000:02:00.0: Adding to iommu group 12
[    2.913964] pci 0000:03:00.0: Adding to iommu group 13
[    2.913986] pci 0000:04:00.0: Adding to iommu group 14
[    2.914007] pci 0000:04:01.0: Adding to iommu group 15
[    2.914029] pci 0000:04:02.0: Adding to iommu group 16
[    2.914050] pci 0000:04:03.0: Adding to iommu group 17
[    2.914083] pci 0000:05:00.0: Adding to iommu group 18
[    2.914114] pci 0000:07:00.0: Adding to iommu group 19
[    2.914123] pci 0000:6e:00.0: Adding to iommu group 20

dmesg | grep -i vfio
Code: 
[    3.337610] VFIO - User Level meta-driver version: 0.3
[    3.340536] vfio_pci: add [8086:43b8[ffffffff:ffffffff]] class 0x000000/00000000
[    3.340539] vfio_pci: add [8086:1136[ffffffff:ffffffff]] class 0x000000/00000000
[    3.340542] vfio_pci: add [8086:1137[ffffffff:ffffffff]] class 0x000000/00000000
[    3.340544] vfio_pci: add [8086:1138[ffffffff:ffffffff]] class 0x000000/00000000
[ 8407.286538] vfio-pci: probe of 0000:00:1c.0 failed with error -22

ls -l /dev/vfio
Code: 
total 0
crw------- 1 root root 242,   0 Jun 15 13:37 18
crw------- 1 root root 242,   1 Jun 15 13:37 19
drwxr-xr-x 2 root root       80 Jun 15 13:37 devices
crw-rw-rw- 1 root root  10, 196 Jun 15 13:37 vfio

lspci -t
Code: 
-[0000:00]-+-00.0
           +-01.0-[01]----00.0
           +-02.0
           +-06.0-[02]----00.0
           +-08.0
           +-14.0
           +-14.2
           +-16.0
           +-17.0
           +-1c.0-[03-6d]----00.0-[04-08]--+-00.0-[05]----00.0
           |                               +-01.0-[06]--
           |                               +-02.0-[07]----00.0
           |                               \-03.0-[08]--
           +-1c.4-[6e]----00.0
           +-1f.0
           +-1f.3
           +-1f.4
           \-1f.5

/sys/bus/pci/devices/0000:00:1c.0/driver_override
Code: 
vfio-pci
[/CODE]
 
Last edited:
lspci -nnPPk
Code: 
00:00.0 Host bridge [0600]: Intel Corporation Device [8086:4c53] (rev 01)
        DeviceName: Onboard - Other
        Subsystem: Micro-Star International Co., Ltd. [MSI] Device [1462:7d04]
        Kernel driver in use: icl_uncore
00:01.0 PCI bridge [0604]: Intel Corporation Device [8086:4c01] (rev 01)
        Subsystem: Micro-Star International Co., Ltd. [MSI] Device [1462:7d04]
        Kernel driver in use: pcieport
00:02.0 VGA compatible controller [0300]: Intel Corporation RocketLake-S GT1 [UHD Graphics 750] [8086:4c8a] (rev 04)
        DeviceName: Onboard - Video
        Subsystem: Micro-Star International Co., Ltd. [MSI] RocketLake-S GT1 [UHD Graphics 750] [1462:7d04]
        Kernel driver in use: i915
        Kernel modules: i915, xe
00:06.0 PCI bridge [0604]: Intel Corporation Device [8086:4c09] (rev 01)
        Kernel driver in use: pcieport
00:08.0 System peripheral [0880]: Intel Corporation Device [8086:4c11] (rev 01)
        DeviceName: Onboard - Other
        Subsystem: Micro-Star International Co., Ltd. [MSI] Device [1462:7d04]
00:14.0 USB controller [0c03]: Intel Corporation Tiger Lake-H USB 3.2 Gen 2x1 xHCI Host Controller [8086:43ed] (rev 11)
        DeviceName: Onboard - Other
        Subsystem: Micro-Star International Co., Ltd. [MSI] Tiger Lake-H USB 3.2 Gen 2x1 xHCI Host Controller [1462:7d04]
        Kernel driver in use: xhci_hcd
        Kernel modules: xhci_pci
00:14.2 RAM memory [0500]: Intel Corporation Tiger Lake-H Shared SRAM [8086:43ef] (rev 11)
        DeviceName: Onboard - Other
00:16.0 Communication controller [0780]: Intel Corporation Tiger Lake-H Management Engine Interface [8086:43e0] (rev 11)
        DeviceName: Onboard - Other
        Subsystem: Micro-Star International Co., Ltd. [MSI] Tiger Lake-H Management Engine Interface [1462:7d04]
        Kernel driver in use: mei_me
        Kernel modules: mei_me
00:17.0 SATA controller [0106]: Intel Corporation Device [8086:43d2] (rev 11)
        DeviceName: Onboard - SATA
        Subsystem: Micro-Star International Co., Ltd. [MSI] Device [1462:7d04]
        Kernel driver in use: ahci
        Kernel modules: ahci
00:1c.0 PCI bridge [0604]: Intel Corporation Tiger Lake-H PCIe Root Port [8086:43b8] (rev 11)
        Subsystem: Micro-Star International Co., Ltd. [MSI] Tiger Lake-H PCIe Root Port [1462:7d04]
        Kernel driver in use: pcieport
00:1c.4 PCI bridge [0604]: Intel Corporation Tiger Lake-H PCI Express Root Port #5 [8086:43bc] (rev 11)
        Subsystem: Micro-Star International Co., Ltd. [MSI] Tiger Lake-H PCI Express Root Port [1462:7d04]
        Kernel driver in use: pcieport
00:1f.0 ISA bridge [0601]: Intel Corporation Z590 LPC/eSPI Controller [8086:4385] (rev 11)
        DeviceName: Onboard - Other
        Subsystem: Micro-Star International Co., Ltd. [MSI] Z590 LPC/eSPI Controller [1462:7d04]
00:1f.3 Audio device [0403]: Intel Corporation Tiger Lake-H HD Audio Controller [8086:43c8] (rev 11)
        DeviceName: Onboard - Sound
        Subsystem: Micro-Star International Co., Ltd. [MSI] Tiger Lake-H HD Audio Controller [1462:7d04]
        Kernel driver in use: snd_hda_intel
        Kernel modules: snd_hda_intel, snd_sof_pci_intel_tgl
00:1f.4 SMBus [0c05]: Intel Corporation Tiger Lake-H SMBus Controller [8086:43a3] (rev 11)
        DeviceName: Onboard - Other
        Subsystem: Micro-Star International Co., Ltd. [MSI] Tiger Lake-H SMBus Controller [1462:7d04]
        Kernel driver in use: i801_smbus
        Kernel modules: i2c_i801
00:1f.5 Serial bus controller [0c80]: Intel Corporation Tiger Lake-H SPI Controller [8086:43a4] (rev 11)
        DeviceName: Onboard - Other
        Subsystem: Micro-Star International Co., Ltd. [MSI] Tiger Lake-H SPI Controller [1462:7d04]
        Kernel driver in use: intel-spi
        Kernel modules: spi_intel_pci
00:01.0/01:00.0 Mass storage controller [0180]: Device [ace1:0018] (rev 01)
        Subsystem: Device [ace1:0018]
00:06.0/02:00.0 Non-Volatile memory controller [0108]: Seagate Technology PLC FireCuda 530 SSD [1bb1:5018] (rev 01)
        Subsystem: Seagate Technology PLC FireCuda 530 SSD [1bb1:5018]
        Kernel driver in use: nvme
        Kernel modules: nvme
00:1c.0/03:00.0 PCI bridge [0604]: Intel Corporation Thunderbolt 4 Bridge [Maple Ridge 4C 2020] [8086:1136] (rev 02)
        Subsystem: Intel Corporation Thunderbolt 4 Bridge [Maple Ridge 4C 2020] [8086:0000]
        Kernel driver in use: pcieport
00:1c.0/03:00.0/04:00.0 PCI bridge [0604]: Intel Corporation Thunderbolt 4 Bridge [Maple Ridge 4C 2020] [8086:1136] (rev 02)
        Subsystem: Intel Corporation Thunderbolt 4 Bridge [Maple Ridge 4C 2020] [8086:0000]
        Kernel driver in use: pcieport
00:1c.0/03:00.0/04:01.0 PCI bridge [0604]: Intel Corporation Thunderbolt 4 Bridge [Maple Ridge 4C 2020] [8086:1136] (rev 02)
        Subsystem: Intel Corporation Thunderbolt 4 Bridge [Maple Ridge 4C 2020] [8086:0000]
        Kernel driver in use: pcieport
00:1c.0/03:00.0/04:02.0 PCI bridge [0604]: Intel Corporation Thunderbolt 4 Bridge [Maple Ridge 4C 2020] [8086:1136] (rev 02)
        Subsystem: Intel Corporation Thunderbolt 4 Bridge [Maple Ridge 4C 2020] [8086:0000]
        Kernel driver in use: pcieport
00:1c.0/03:00.0/04:03.0 PCI bridge [0604]: Intel Corporation Thunderbolt 4 Bridge [Maple Ridge 4C 2020] [8086:1136] (rev 02)
        Subsystem: Intel Corporation Thunderbolt 4 Bridge [Maple Ridge 4C 2020] [8086:0000]
        Kernel driver in use: pcieport
00:1c.0/03:00.0/04:00.0/05:00.0 USB controller [0c03]: Intel Corporation Thunderbolt 4 NHI [Maple Ridge 4C 2020] [8086:1137]
        Subsystem: Intel Corporation Thunderbolt 4 NHI [Maple Ridge 4C 2020] [8086:0000]
        Kernel driver in use: vfio-pci
        Kernel modules: thunderbolt
00:1c.0/03:00.0/04:02.0/07:00.0 USB controller [0c03]: Intel Corporation Thunderbolt 4 USB Controller [Maple Ridge 4C 2020] [8086:1138]
        Kernel driver in use: vfio-pci
        Kernel modules: xhci_pci
00:1c.4/6e:00.0 Ethernet controller [0200]: Intel Corporation Ethernet Controller I225-V [8086:15f3] (rev 03)
        Subsystem: Micro-Star International Co., Ltd. [MSI] Ethernet Controller I225-V [1462:7d04]
        Kernel driver in use: igc
        Kernel modules: igc

for d in /sys/kernel/iommu_groups/*/devices/*; do echo "$d"; lspci -nns "${d##*/}"; done | less
Code: 
/sys/kernel/iommu_groups/0/devices/0000:00:02.0
00:02.0 VGA compatible controller [0300]: Intel Corporation RocketLake-S GT1 [UHD Graphics 750] [8086:4c8a] (rev 04)
/sys/kernel/iommu_groups/10/devices/0000:00:1f.0
00:1f.0 ISA bridge [0601]: Intel Corporation Z590 LPC/eSPI Controller [8086:4385] (rev 11)
/sys/kernel/iommu_groups/10/devices/0000:00:1f.3
00:1f.3 Audio device [0403]: Intel Corporation Tiger Lake-H HD Audio Controller [8086:43c8] (rev 11)
/sys/kernel/iommu_groups/10/devices/0000:00:1f.4
00:1f.4 SMBus [0c05]: Intel Corporation Tiger Lake-H SMBus Controller [8086:43a3] (rev 11)
/sys/kernel/iommu_groups/10/devices/0000:00:1f.5
00:1f.5 Serial bus controller [0c80]: Intel Corporation Tiger Lake-H SPI Controller [8086:43a4] (rev 11)
/sys/kernel/iommu_groups/11/devices/0000:01:00.0
01:00.0 Mass storage controller [0180]: Device [ace1:0018] (rev 01)
/sys/kernel/iommu_groups/12/devices/0000:02:00.0
02:00.0 Non-Volatile memory controller [0108]: Seagate Technology PLC FireCuda 530 SSD [1bb1:5018] (rev 01)
/sys/kernel/iommu_groups/13/devices/0000:03:00.0
03:00.0 PCI bridge [0604]: Intel Corporation Thunderbolt 4 Bridge [Maple Ridge 4C 2020] [8086:1136] (rev 02)
/sys/kernel/iommu_groups/14/devices/0000:04:00.0
04:00.0 PCI bridge [0604]: Intel Corporation Thunderbolt 4 Bridge [Maple Ridge 4C 2020] [8086:1136] (rev 02)
/sys/kernel/iommu_groups/15/devices/0000:04:01.0
04:01.0 PCI bridge [0604]: Intel Corporation Thunderbolt 4 Bridge [Maple Ridge 4C 2020] [8086:1136] (rev 02)
/sys/kernel/iommu_groups/16/devices/0000:04:02.0
04:02.0 PCI bridge [0604]: Intel Corporation Thunderbolt 4 Bridge [Maple Ridge 4C 2020] [8086:1136] (rev 02)
/sys/kernel/iommu_groups/17/devices/0000:04:03.0
04:03.0 PCI bridge [0604]: Intel Corporation Thunderbolt 4 Bridge [Maple Ridge 4C 2020] [8086:1136] (rev 02)
/sys/kernel/iommu_groups/18/devices/0000:05:00.0
05:00.0 USB controller [0c03]: Intel Corporation Thunderbolt 4 NHI [Maple Ridge 4C 2020] [8086:1137]
/sys/kernel/iommu_groups/19/devices/0000:07:00.0
07:00.0 USB controller [0c03]: Intel Corporation Thunderbolt 4 USB Controller [Maple Ridge 4C 2020] [8086:1138]
/sys/kernel/iommu_groups/1/devices/0000:00:00.0
00:00.0 Host bridge [0600]: Intel Corporation Device [8086:4c53] (rev 01)
/sys/kernel/iommu_groups/20/devices/0000:6e:00.0
6e:00.0 Ethernet controller [0200]: Intel Corporation Ethernet Controller I225-V [8086:15f3] (rev 03)
/sys/kernel/iommu_groups/2/devices/0000:00:01.0
00:01.0 PCI bridge [0604]: Intel Corporation Device [8086:4c01] (rev 01)
/sys/kernel/iommu_groups/3/devices/0000:00:06.0
00:06.0 PCI bridge [0604]: Intel Corporation Device [8086:4c09] (rev 01)
/sys/kernel/iommu_groups/4/devices/0000:00:08.0
00:08.0 System peripheral [0880]: Intel Corporation Device [8086:4c11] (rev 01)
/sys/kernel/iommu_groups/5/devices/0000:00:14.0
00:14.0 USB controller [0c03]: Intel Corporation Tiger Lake-H USB 3.2 Gen 2x1 xHCI Host Controller [8086:43ed] (rev 11)
/sys/kernel/iommu_groups/5/devices/0000:00:14.2
00:14.2 RAM memory [0500]: Intel Corporation Tiger Lake-H Shared SRAM [8086:43ef] (rev 11)
/sys/kernel/iommu_groups/6/devices/0000:00:16.0
00:16.0 Communication controller [0780]: Intel Corporation Tiger Lake-H Management Engine Interface [8086:43e0] (rev 11)
/sys/kernel/iommu_groups/7/devices/0000:00:17.0
00:17.0 SATA controller [0106]: Intel Corporation Device [8086:43d2] (rev 11)
/sys/kernel/iommu_groups/8/devices/0000:00:1c.0
00:1c.0 PCI bridge [0604]: Intel Corporation Tiger Lake-H PCIe Root Port [8086:43b8] (rev 11)
/sys/kernel/iommu_groups/9/devices/0000:00:1c.4
00:1c.4 PCI bridge [0604]: Intel Corporation Tiger Lake-H PCI Express Root Port #5 [8086:43bc] (rev 11)

find /sys/kernel/iommu_groups/ -type l
Code: 
/sys/kernel/iommu_groups/17/devices/0000:04:03.0
/sys/kernel/iommu_groups/7/devices/0000:00:17.0
/sys/kernel/iommu_groups/15/devices/0000:04:01.0
/sys/kernel/iommu_groups/5/devices/0000:00:14.2
/sys/kernel/iommu_groups/5/devices/0000:00:14.0
/sys/kernel/iommu_groups/13/devices/0000:03:00.0
/sys/kernel/iommu_groups/3/devices/0000:00:06.0
/sys/kernel/iommu_groups/11/devices/0000:01:00.0
/sys/kernel/iommu_groups/1/devices/0000:00:00.0
/sys/kernel/iommu_groups/18/devices/0000:05:00.0
/sys/kernel/iommu_groups/8/devices/0000:00:1c.0
/sys/kernel/iommu_groups/16/devices/0000:04:02.0
/sys/kernel/iommu_groups/6/devices/0000:00:16.0
/sys/kernel/iommu_groups/14/devices/0000:04:00.0
/sys/kernel/iommu_groups/4/devices/0000:00:08.0
/sys/kernel/iommu_groups/12/devices/0000:02:00.0
/sys/kernel/iommu_groups/2/devices/0000:00:01.0
/sys/kernel/iommu_groups/20/devices/0000:6e:00.0
/sys/kernel/iommu_groups/10/devices/0000:00:1f.0
/sys/kernel/iommu_groups/10/devices/0000:00:1f.5
/sys/kernel/iommu_groups/10/devices/0000:00:1f.3
/sys/kernel/iommu_groups/10/devices/0000:00:1f.4
/sys/kernel/iommu_groups/0/devices/0000:00:02.0
/sys/kernel/iommu_groups/19/devices/0000:07:00.0
/sys/kernel/iommu_groups/9/devices/0000:00:1c.4

so in the end the pcie and the thunderbolt are
Code: 
pcie 0000:00:1c.0
thunderbolt bridge 0000:03:00.0
thunderbolt bridge 0000:04:00.0
thunderbolt bridge 0000:04:01.0
thunderbolt bridge 0000:04:02.0
thunderbolt bridge 0000:04:03.0
thunderbolt nhi 0000:05:00.0
thunderbolt usb 0000:07:00.0



and this is the vm config a windows 10 /etc/pve/qemu-server/100.conf
Code: 
#args%3A -device qemu-xhci,id=xhci1 -device usb-host,bus=xhci1.0,hostbus=2,hostport=1,id=Device_1 -device usb-host,bus=xhci1.0,hostbus=1,hostport=1.7,id=Device_2
agent: 1
args: -cpu 'host,-hypervisor,kvm=off'
audio0: device=ich9-intel-hda,driver=none
bios: ovmf
boot: order=scsi0;sata0
cores: 2
cpu: host,flags=+hv-evmcs
efidisk0: local-lvm:vm-100-disk-0,efitype=4m,pre-enrolled-keys=1,size=4M
hostpci0: 0000:00:1c.0,pcie=1
hostpci1: 0000:03:00.0,pcie=1
hostpci2: 0000:04:00.0,pcie=1
hostpci3: 0000:04:01.0,pcie=1
hostpci4: 0000:04:02.0,pcie=1
hostpci5: 0000:04:03.0,pcie=1
hostpci6: 0000:05:00.0,pcie=1
hostpci7: 0000:07:00.0,pcie=1
hotplug: usb
machine: pc-q35-8.1
memory: 8192
meta: creation-qemu=8.1.2,ctime=1701594220
name: win10base
net0: virtio=BC:24:11:CB:47:85,bridge=vmbr0
numa: 0
ostype: win10
sata0: none,media=cdrom
scsi0: local-lvm:vm-100-disk-1,cache=writeback,discard=on,size=64G,ssd=1
scsihw: virtio-scsi-pci
smbios1: uuid=85095253-388e-46d7-a52e-c5ac1df333f3
sockets: 2
vga: std,memory=256
vmgenid: dd77a721-0440-47a8-a25a-5608676959ac
it giveme error when i try to run it
Code: 
kvm: -device vfio-pci,host=0000:00:1c.0,id=hostpci0,bus=ich9-pcie-port-1,addr=0x0: vfio 0000:00:1c.0: failed to open /dev/vfio/8: No such file or directory
TASK ERROR: start failed: QEMU exited with code 1

if i map the 0000:00:1c it maps 0000:00:1c.0 and 0000:00:1c.4 Tyger Lake-H pci express root port #5
happend the same

i try
Code: 
echo 0000:00:1c.0 | sudo tee /sys/bus/pci/devices/0000:00:1c.0/driver/unbind
0000:00:1c.0
and then
Code: 
echo 8086 43b8 | sudo tee /sys/bus/pci/drivers/vfio-pci/new_id
tee: /sys/bus/pci/drivers/vfio-pci/new_id: File exists
echo 0000:00:1c.0 | sudo tee /sys/bus/pci/drivers/vfio-pci/bind
tee: /sys/bus/pci/drivers/vfio-pci/bind: Invalid argument


so im out of ideas if anyone could helpme
 
also looking for this. i wasn't able to get this working. i'm trying to run a daily driver ubuntu QEMU in proxmox but I wanted to get single cable so everything is going through single thunderbolt connection. (7900 xtx going in to Asus ProArt z790 DP ins)
 
  • Like
Reactions: mikeone
You must log in or register to reply here.
Top Bottom