Problem Passing Thunderbolt Titan Ridge PCIe Card

iGPU · May 10, 2020

I'm having difficulty in passing a Gigabyte (GB) Thunderbolt Titan Ridge PCIe card.

The machine is a GB TRX40 Designare with an add-on TB3 Titan Ridge PCIe card. The guest OS running under Proxmox is macOS Mojave. I've managed to pass GPU, add-on Aquantia Ethernet card, and various other internal controllers (SATA, Ethernet and BT/Wifi) successfully and these are running well under macOS. However, I cannot get the Thunderbolt card to pass; well, not all parts, that is.

Before describing more details, I've tried the card in various slots and from internet searches the optimum seems to be slot-2 (nevertheless, I see similar results to those shown below for all tested slots).

To show the distribution of the Thunderbolt card, you can see the various sections, which are from 01:00 to 0d:00, with an lspci command:

Code:

Addresses Before TB3 Section:
00:00.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Root Complex [1022:1480]
        Subsystem: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Root Complex [1022:1480]
00:00.2 IOMMU [0806]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse IOMMU [1022:1481]
        Subsystem: Advanced Micro Devices, Inc. [AMD] Starship/Matisse IOMMU [1022:1481]
00:01.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
00:01.3 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge [1022:1483]
        Kernel driver in use: pcieport
00:02.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
00:03.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
00:04.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
00:05.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
00:07.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
00:07.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] [1022:1484]
        Kernel driver in use: pcieport
00:08.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
00:08.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] [1022:1484]
        Kernel driver in use: pcieport
00:14.0 SMBus [0c05]: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller [1022:790b] (rev 61)
        Subsystem: Gigabyte Technology Co., Ltd FCH SMBus Controller [1458:5001]
        Kernel modules: i2c_piix4, sp5100_tco
00:14.3 ISA bridge [0601]: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge [1022:790e] (rev 51)
        Subsystem: Gigabyte Technology Co., Ltd FCH LPC Bridge [1458:5001]
00:18.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship Device 24; Function 0 [1022:1490]
00:18.1 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship Device 24; Function 1 [1022:1491]
00:18.2 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship Device 24; Function 2 [1022:1492]
00:18.3 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship Device 24; Function 3 [1022:1493]
        Kernel driver in use: k10temp
        Kernel modules: k10temp
00:18.4 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship Device 24; Function 4 [1022:1494]
00:18.5 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship Device 24; Function 5 [1022:1495]
00:18.6 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship Device 24; Function 6 [1022:1496]
00:18.7 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship Device 24; Function 7 [1022:1497]


TB3 Address Sections:
01:00.0 PCI bridge [0604]: Intel Corporation JHL7540 Thunderbolt 3 Bridge [Titan Ridge 4C 2018] [8086:15ea] (rev 06)
        Kernel driver in use: pcieport
02:00.0 PCI bridge [0604]: Intel Corporation JHL7540 Thunderbolt 3 Bridge [Titan Ridge 4C 2018] [8086:15ea] (rev 06)
        Kernel driver in use: pcieport
02:01.0 PCI bridge [0604]: Intel Corporation JHL7540 Thunderbolt 3 Bridge [Titan Ridge 4C 2018] [8086:15ea] (rev 06)
        Kernel driver in use: pcieport
02:02.0 PCI bridge [0604]: Intel Corporation JHL7540 Thunderbolt 3 Bridge [Titan Ridge 4C 2018] [8086:15ea] (rev 06)
        Kernel driver in use: pcieport
02:04.0 PCI bridge [0604]: Intel Corporation JHL7540 Thunderbolt 3 Bridge [Titan Ridge 4C 2018] [8086:15ea] (rev 06)
        Kernel driver in use: pcieport
03:00.0 System peripheral [0880]: Intel Corporation JHL7540 Thunderbolt 3 NHI [Titan Ridge 4C 2018] [8086:15eb] (rev 06)
        Subsystem: Device [2222:1111]
        Kernel driver in use: vfio-pci
        Kernel modules: thunderbolt
0d:00.0 USB controller [0c03]: Intel Corporation JHL7540 Thunderbolt 3 USB Controller [Titan Ridge 4C 2018] [8086:15ec] (rev 06)
        Subsystem: Device [2222:1111]
        Kernel driver in use: vfio-pci


BTW, the vfio.conf file contains the TB sections: "options vfio-pci ids=8086:15ea,8086:15eb,8086:15ec"

From the above, you can see that 03:00 and 0d:00 have their drivers correctly substituted by vfio-pci (the original drivers were thunderbolt and xhci_hcd). And these 2 sections are passed through to the macOS system. However, the whole TB3 'tree' does not appear in the macOS as the 01:00 and 02:00 sections are missing. In macOS, these sections seems to correspond to internal TB3 bridges. Accordingly, when these sections are missing, Thunderbolt does not work. The basic problem, then, is that the whole device is not being passed.

If the tree command is run, the following is seen:

Code:

\-[0000:00]-+-00.0  Advanced Micro Devices, Inc. [AMD] Starship/Matisse Root Complex
                     +-00.2  Advanced Micro Devices, Inc. [AMD] Starship/Matisse IOMMU
                     +-01.0  Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
                     +-01.3-[01-16]----00.0-[02-16]--+-00.0-[03]----00.0  Intel Corporation JHL7540 Thunderbolt 3 NHI [Titan Ridge 4C 2018]
                     |                               +-01.0-[04-0c]--
                     |                               +-02.0-[0d]----00.0  Intel Corporation JHL7540 Thunderbolt 3 USB Controller [Titan Ridge 4C 2018]
                     |                               \-04.0-[0e-16]--

The sections that are not passed, 01:00.0 and 02:00.0x.0, have pcieport drivers. Blacklisting or not blacklisting the pcieport driver has no effect. In fact, if either 01:00 or 02:00 are entered into the VM using "hostpci9: 1:00.0,pcie=1" (±rombar), then the VM fails to run, generating an error that says:

vm: -device vfio-pci,host=0000:01:00.0,id=hostpci9,bus=ich9-pcie-port-10,addr=0x0: vfio 0000:01:00.0: failed to open /dev/vfio/12: No such file or directory
TASK ERROR: start failed: QEMU exited with code 1

The file named "12" that seems to be missing in /dev/vfio/, is the IOMMU group for 1:00.0. And if 02:00.0 is attempted to be passed, it gives a similar error saying file "13" is missing from that same directory, which is 02:00.0's IOMMU group number. Meanwhile, the sections that are passed, have files in this directory for their IOMMU groups, namely 17 and 18. The files presently in this directory are: 17 18 20 34 35 37 57 59 60 63 64 65 67 78 vfio.

The complete TB3 IOMMU groups can be seen when "ls /sys/kernel/iommu_groups/*/devices" is run:

Code:

/sys/kernel/iommu_groups/12/devices:  0000:01:00.0
/sys/kernel/iommu_groups/13/devices:  0000:02:00.0
/sys/kernel/iommu_groups/14/devices:  0000:02:01.0
/sys/kernel/iommu_groups/15/devices:  0000:02:02.0
/sys/kernel/iommu_groups/16/devices:  0000:02:04.0
/sys/kernel/iommu_groups/17/devices:  0000:03:00.0
/sys/kernel/iommu_groups/18/devices:  0000:0d:00.0

I'd truly appreciate any help in getting the whole TB device passed. Thanks!

Stefan_R · May 13, 2020

Passing through an entire TB3 card seems like a tall order... Certainly an interesting concept, but I see several problems which make me think this is not really possible ATM:

The TB3 protocol uses not just PCIe, but also the seperate TB3 header, which, AFAIK, cannot be passed through
At it's core, thunderbolt is just "external PCIe", so the controller itself would act as a bridge in the topology (that's why you're seeing the 'pcieport' driver being applied). vfio-pci is usually only capable of passing through leaf devices, i.e. a GPU connected via TB3 might be able to work fine in a VM, but the entire controller probably won't
TB3 is designed with hot-plug in mind, and I while hotplugging emulated PCIe devices into a VM is certainly possible, I don't think this would work with such a "double-layer" hotplug (i.e. first the host, then the guest, the device is effectively hotplugged twice)

Personally never heard of anyone attempting this, let alone succeeding, but if you manage to get it to work I'd be very glad to hear your report

iGPU · May 14, 2020

Stefan_R said:
Passing through an entire TB3 card seems like a tall order... Certainly an interesting concept, but I see several problems which make me think this is not really possible ATM:

The TB3 protocol uses not just PCIe, but also the seperate TB3 header, which, AFAIK, cannot be passed through

At it's core, thunderbolt is just "external PCIe", so the controller itself would act as a bridge in the topology (that's why you're seeing the 'pcieport' driver being applied). vfio-pci is usually only capable of passing through leaf devices, i.e. a GPU connected via TB3 might be able to work fine in a VM, but the entire controller probably won't

TB3 is designed with hot-plug in mind, and I while hotplugging emulated PCIe devices into a VM is certainly possible, I don't think this would work with such a "double-layer" hotplug (i.e. first the host, then the guest, the device is effectively hotplugged twice)

Personally never heard of anyone attempting this, let alone succeeding, but if you manage to get it to work I'd be very glad to hear your report

Thanks for reply. Disappointing news, but I suspected as much.

With some effort, I have the majority of a TB tree appearing on the Mac side (the card is flashed with modified firmware), but I cannot get an SSDT to be applied as the TB tree is appearing nested in parallel with the NVMe drive and one of the SATA controllers that are passed through.

I've checked various positions for the NVMe drives (one drive each for macOS and Proxmox) in combination with the TB card position. However, it doesn't seem to matter where they're placed: they get grouped the same on the Mac side and I cannot apply the SSDT. If the SSDT cannot be applied, TB won't work under macOS.

In the attached image, the TB tree is outlined in red on the left. In the top red outline, can be seen only the NHI and USB sections that were passed. Normally, one would see 4 or 5 more entries for PCI bridge sections, which correlate with DSB0-DSB4 nodes. But despite the missing parts, the right most red outline shows a normal TB bus being created by the macOS. (The GB4 window was placed simply to show the processor wass indeed the AMD 3970X.)

smith000 · Mar 23, 2023

iGPU said:
Thanks for reply. Disappointing news, but I suspected as much.

With some effort, I have the majority of a TB tree appearing on the Mac side (the card is flashed with modified firmware), but I cannot get an SSDT to be applied as the TB tree is appearing nested in parallel with the NVMe drive and one of the SATA controllers that are passed through.

I've checked various positions for the NVMe drives (one drive each for macOS and Proxmox) in combination with the TB card position. However, it doesn't seem to matter where they're placed: they get grouped the same on the Mac side and I cannot apply the SSDT. If the SSDT cannot be applied, TB won't work under macOS.

In the attached image, the TB tree is outlined in red on the left. In the top red outline, can be seen only the NHI and USB sections that were passed. Normally, one would see 4 or 5 more entries for PCI bridge sections, which correlate with DSB0-DSB4 nodes. But despite the missing parts, the right most red outline shows a normal TB bus being created by the macOS. (The GB4 window was placed simply to show the processor wass indeed the AMD 3970X.)

I'd be interested to see your vfio and guest config.

There's 3 PCI devices that it looks like are passable with VFIO, but one of them refuses to pass:


root@pve:~# qm start 101 kvm: -device vfio-pci,host=0000:0f:00.0,id=hostpci5,bus=ich9-pcie-port-6,addr=0x0: vfio 0000:0f:00.0: error getting device from group 29: Invalid argument Verify all devices in group 29 are bound to vfio-<bus> or pci-stub and not already in use

The three PCI devices I have are:


12:00.0 USB controller [0c03]: Intel Corporation Thunderbolt 4 USB Controller [Goshen Ridge 2020] [8086:0b27] (rev 02)    Subsystem: Intel Corporation Thunderbolt 4 USB Controller [Goshen Ridge 2020] [8086:0000]    Kernel driver in use: xhci_hcd    Kernel modules: xhci_pci    

06:00.0 System peripheral [0880]: Intel Corporation JHL7540 Thunderbolt 3 NHI [Titan Ridge 4C 2018] [8086:15eb] (rev 06)    Subsystem: Gigabyte Technology Co., Ltd JHL7540 Thunderbolt 3 NHI [Titan Ridge 4C 2018] [1458:a207]    Kernel driver in use: vfio-pci    Kernel modules: thunderbolt 

0f:00.0 USB controller [0c03]: Intel Corporation JHL7540 Thunderbolt 3 USB Controller [Titan Ridge 4C 2018] [8086:15ec] (rev 06)    Subsystem: Gigabyte Technology Co., Ltd JHL7540 Thunderbolt 3 USB Controller [Titan Ridge 4C 2018] [1458:a207]    Kernel driver in use: vfio-pci    Kernel modules: xhci_pci

Another odd quirk is that the 12:00.0 device, the thunderbolt 4 controller, won't even show up in the device list unless there's something plugged into the thunderbolt port that supports USB.

I've successfully gotten the guest to see the thunderbolt dock attached. Both windows (thunderbolt control center) and windows (boltctl) see it just fine.

I haven't tried to do this on bare metal but I don't have another AMD box anyway.

I'd love to see your vfio & guest config you used to get this working. Again, it's specifically the USB2 that seems to be the holdout. Unfortunately that's the meat of what the dock does - my whole reason for this exercise.

-JS

deliad · Jul 6, 2023

smith000 said:
I'd be interested to see your vfio and guest config.

There's 3 PCI devices that it looks like are passable with VFIO, but one of them refuses to pass:

root@pve:~# qm start 101 kvm: -device vfio-pci,host=0000:0f:00.0,id=hostpci5,bus=ich9-pcie-port-6,addr=0x0: vfio 0000:0f:00.0: error getting device from group 29: Invalid argument Verify all devices in group 29 are bound to vfio-<bus> or pci-stub and not already in use

The three PCI devices I have are:

12:00.0 USB controller [0c03]: Intel Corporation Thunderbolt 4 USB Controller [Goshen Ridge 2020] [8086:0b27] (rev 02) Subsystem: Intel Corporation Thunderbolt 4 USB Controller [Goshen Ridge 2020] [8086:0000] Kernel driver in use: xhci_hcd Kernel modules: xhci_pci 06:00.0 System peripheral [0880]: Intel Corporation JHL7540 Thunderbolt 3 NHI [Titan Ridge 4C 2018] [8086:15eb] (rev 06) Subsystem: Gigabyte Technology Co., Ltd JHL7540 Thunderbolt 3 NHI [Titan Ridge 4C 2018] [1458:a207] Kernel driver in use: vfio-pci Kernel modules: thunderbolt 0f:00.0 USB controller [0c03]: Intel Corporation JHL7540 Thunderbolt 3 USB Controller [Titan Ridge 4C 2018] [8086:15ec] (rev 06) Subsystem: Gigabyte Technology Co., Ltd JHL7540 Thunderbolt 3 USB Controller [Titan Ridge 4C 2018] [1458:a207] Kernel driver in use: vfio-pci Kernel modules: xhci_pci

Another odd quirk is that the 12:00.0 device, the thunderbolt 4 controller, won't even show up in the device list unless there's something plugged into the thunderbolt port that supports USB.

I've successfully gotten the guest to see the thunderbolt dock attached. Both windows (thunderbolt control center) and windows (boltctl) see it just fine.

I haven't tried to do this on bare metal but I don't have another AMD box anyway.

I'd love to see your vfio & guest config you used to get this working. Again, it's specifically the USB2 that seems to be the holdout. Unfortunately that's the meat of what the dock does - my whole reason for this exercise.

-JS

did you solved the problem?

i have ASrock X570 Creator with TB3 and has the same issues

Docop2 · Jul 7, 2023

tb is a protocol that extend pci-e lane. when you do lspci, you do see cpu. do you try to pass the cpu ? tb is part of, so only external device plugged will be compatible. Amd system are worst and so thoses asus plug in card. asrock are quite fine, while i haven't tested this one. And different tb unit device cannot be pass or pass half their function. But on prox 7.1 and 6.4 something, a nuc with a quadro was soo easy to pass direct to win10 vm all up, no mod in the windows vm. but new prox version kill that sweet easy work. Lot of try to do, you have to try on your own machine in order to know if all do work based on your used case. And to save couple days of fight, always install win10 on the system. no linux, no prox.. win and confirm the tb unit do work properly as cabling cause many issues as well. Like aoc cable .. they don't survive long.

Search

Search

Problem Passing Thunderbolt Titan Ridge PCIe Card

iGPU

New Member

Stefan_R

Proxmox Retired Staff

iGPU

New Member

Attachments

smith000

Member

deliad

New Member

Docop2

Member