Thunderbolt passthrough partially working need help

M

mikeone

New Member
Proxmox Subscriber
Sep 17, 2023
17
1
3
Hi there,

I have just finished installing my new TR server with the Asus WRX90 Board, I have a perfectly working Win10 VM with GPU passthrough, however I am unable to pass on the thunderbolt controller in order to use a Thunderbolt dock, the thunderbolt dock is detected and on, I can see it when typing "lspci" but when I pass it to the VM it's crash and won't start.

I have added the device ID in the vfio file, but I did not solve my issue, did any one stumble upon a similar case ?

this is the output of "dmesg | grep -i vfio"

Code: 
root@pve:~#dmesg | grep -i vfio
[    9.151809] VFIO - User Level meta-driver version: 0.3
[   18.573883] vfio-pci 0000:c1:00.0: vgaarb: deactivate vga console
[   18.573886] vfio-pci 0000:c1:00.0: vgaarb: VGA decodes changed: olddecodes=io+mem,decodes=io+mem:owns=none
[   18.574365] vfio-pci 0000:c1:00.0: vgaarb: VGA decodes changed: olddecodes=io+mem,decodes=io+mem:owns=none
[   18.574411] vfio-pci 0000:c1:00.0: vgaarb: deactivate vga console
[   18.574413] vfio-pci 0000:c1:00.0: vgaarb: VGA decodes changed: olddecodes=io+mem,decodes=io+mem:owns=none
[  106.090514] vfio-pci 0000:5e:00.0: Unable to change power state from unknown to D0, device inaccessible
[  106.090549] vfio-pci 0000:5e:00.0: Unable to change power state from D3cold to D0, device inaccessible
[  106.090730] vfio-pci 0000:5e:00.0: Unable to change power state from D3cold to D0, device inaccessible
[  106.090738] vfio-pci 0000:5e:00.0: Unable to change power state from D3cold to D0, device inaccessible
[  108.321990] vfio-pci 0000:5e:00.0: not ready 1023ms after bus reset; waiting
[  109.408983] vfio-pci 0000:5e:00.0: not ready 2047ms after bus reset; waiting
[  111.520967] vfio-pci 0000:5e:00.0: not ready 4095ms after bus reset; waiting
[  115.808930] vfio-pci 0000:5e:00.0: not ready 8191ms after bus reset; waiting
[  124.513861] vfio-pci 0000:5e:00.0: not ready 16383ms after bus reset; waiting
[  141.409235] vfio-pci 0000:5e:00.0: not ready 32767ms after bus reset; waiting
[  178.272377] vfio-pci 0000:5e:00.0: not ready 65535ms after bus reset; giving up
[  178.276246] vfio-pci 0000:5e:00.0: Unable to change power state from D3cold to D0, device inaccessible
[  252.002106] vfio-pci 0000:5e:00.0: Unable to change power state from D3cold to D0, device inaccessible
[  252.002112] vfio-pci 0000:5e:00.0: Unable to change power state from D3cold to D0, device inaccessible
[  252.002146] vfio-pci 0000:5e:00.0: Unable to change power state from D3cold to D0, device inaccessible
[  252.005957] vfio-pci 0000:5e:00.0: Unable to change power state from D3cold to D0, device inaccessible
[40245.054950] vfio-pci 0000:5e:00.0: Unable to change power state from D3cold to D0, device inaccessible
[40245.054959] vfio-pci 0000:5e:00.0: Unable to change power state from D3cold to D0, device inaccessible
[40245.055050] vfio-pci 0000:5e:00.0: Unable to change power state from D3cold to D0, device inaccessible
[40245.055066] vfio-pci 0000:5e:00.0: Unable to change power state from D3cold to D0, device inaccessible
Thanks for the help
 
Last edited:
Is it possible that it shares an IOMMU group with other devices that are needed by the host? You can check the IOMMU groups via the following bash script:

Code: 
#!/bin/bash
shopt -s nullglob
for g in $(find /sys/kernel/iommu_groups/* -maxdepth 0 -type d | sort -V); do
    echo "IOMMU Group ${g##*/}:"
    for d in $g/devices/*; do
        echo -e "\t$(lspci -nns ${d##*/})"
    done;
done;
 
I have already checked and used also another similar script to have even more details, please see the output below.


root@pve:~# bash script.sh
Group 0: [1022:14a6] e0:00.3 Generic system peripheral [0807] Device 14a6
Group 1: [1022:149f] e0:01.0 Host bridge Device 149f
Group 2: [1022:149f] e0:02.0 Host bridge Device 149f
Group 3: [1022:149f] e0:03.0 Host bridge Device 149f
Group 4: [1022:14a5] [R] e0:03.3 PCI bridge Device 14a5
Group 5: [1022:14a5] [R] e0:03.4 PCI bridge Device 14a5
Group 6: [1022:149f] e0:04.0 Host bridge Device 149f
Group 7: [1022:149f] e0:05.0 Host bridge Device 149f
Group 8: [1022:14aa] [R] e0:05.2 PCI bridge Device 14aa
Group 9: [1022:149f] e0:07.0 Host bridge Device 149f
[1022:14a7] [R] e0:07.1 PCI bridge Device 14a7
[1022:14ac] [R] e5:00.0 Non-Essential Instrumentation [1300] Device 14ac
[1022:14c9] [R] e5:00.4 USB controller Device 14c9
USB: [1d6b:0002] Bus 001 Device 001 Linux Foundation 2.0 root hub
USB: [1d6b:0003] Bus 002 Device 001 Linux Foundation 3.0 root hub
Group 10: [144d:a80c] [R] e1:00.0 Non-Volatile memory controller NVMe SSD Controller S4LV008[Pascal]
Group 11: [144d:a80c] [R] e2:00.0 Non-Volatile memory controller NVMe SSD Controller S4LV008[Pascal]
Group 12: [1a03:1150] [R] e3:00.0 PCI bridge AST1150 PCI-to-PCI Bridge
[1a03:2000] [R] e4:00.0 VGA compatible controller ASPEED Graphics Family
Group 13: [1022:14a6] c0:00.3 Generic system peripheral [0807] Device 14a6
Group 14: [1022:149f] c0:01.0 Host bridge Device 149f
Group 15: [1022:149f] c0:02.0 Host bridge Device 149f
Group 16: [1022:149f] c0:03.0 Host bridge Device 149f
Group 17: [1022:14a5] [R] c0:03.1 PCI bridge Device 14a5
Group 18: [1022:149f] c0:04.0 Host bridge Device 149f
Group 19: [1022:149f] c0:05.0 Host bridge Device 149f
Group 20: [1022:149f] c0:07.0 Host bridge Device 149f
[1022:14a7] [R] c0:07.1 PCI bridge Device 14a7
[1022:14ac] [R] c2:00.0 Non-Essential Instrumentation [1300] Device 14ac
Group 21: [10de:2204] [R] c1:00.0 VGA compatible controller GA102 [GeForce RTX 3090]
[10de:1aef] c1:00.1 Audio device GA102 High Definition Audio Controller
Group 22: [1022:14a6] 00:00.3 Generic system peripheral [0807] Device 14a6
Group 23: [1022:149f] 00:01.0 Host bridge Device 149f
Group 24: [1022:149f] 00:02.0 Host bridge Device 149f
Group 25: [1022:149f] 00:03.0 Host bridge Device 149f
Group 26: [1022:149f] 00:04.0 Host bridge Device 149f
Group 27: [1022:149f] 00:05.0 Host bridge Device 149f
Group 28: [1022:14aa] [R] 00:05.1 PCI bridge Device 14aa
Group 29: [1022:149f] 00:07.0 Host bridge Device 149f
[1022:14a7] [R] 00:07.1 PCI bridge Device 14a7
[1022:14ac] [R] 03:00.0 Non-Essential Instrumentation [1300] Device 14ac
[1022:14c9] [R] 03:00.4 USB controller Device 14c9
USB: [1038:12e0] Bus 003 Device 002 SteelSeries ApS Arctis Nova Pro Wireless
USB: [1d6b:0002] Bus 003 Device 001 Linux Foundation 2.0 root hub
USB: [1d6b:0003] Bus 004 Device 001 Linux Foundation 3.0 root hub
[1022:14ca] [R] 03:00.5 Encryption controller Genoa CCP/PSP 4.0 Device
[1022:14cc] 03:00.7 Audio device Device 14cc
Group 30: [1022:790b] 00:14.0 SMBus FCH SMBus Controller
[1022:790e] 00:14.3 ISA bridge FCH LPC Bridge
Group 31: [1022:14ad] 00:18.0 Host bridge Device 14ad
[1022:14ae] 00:18.1 Host bridge Device 14ae
[1022:14af] 00:18.2 Host bridge Device 14af
[1022:14b0] 00:18.3 Host bridge Device 14b0
[1022:14b1] 00:18.4 Host bridge Device 14b1
[1022:14b2] 00:18.5 Host bridge Device 14b2
[1022:14b3] 00:18.6 Host bridge Device 14b3
[1022:14b4] 00:18.7 Host bridge Device 14b4
Group 32: [8086:15ff] [R] 01:00.0 Ethernet controller Ethernet Controller X710 for 10GBASE-T
Group 33: [8086:15ff] [R] 01:00.1 Ethernet controller Ethernet Controller X710 for 10GBASE-T
Group 34: [1022:14a6] 20:00.3 Generic system peripheral [0807] Device 14a6
Group 35: [1022:149f] 20:01.0 Host bridge Device 149f
Group 36: [1022:14ab] [R] 20:01.1 PCI bridge Device 14ab
Group 37: [1022:149f] 20:02.0 Host bridge Device 149f
Group 38: [1022:149f] 20:03.0 Host bridge Device 149f
Group 39: [1022:14a5] [R] 20:03.1 PCI bridge Device 14a5
Group 40: [1022:14a5] [R] 20:03.3 PCI bridge Device 14a5
Group 41: [1022:14a5] [R] 20:03.4 PCI bridge Device 14a5
Group 42: [1022:149f] 20:04.0 Host bridge Device 149f
Group 43: [1022:149f] 20:05.0 Host bridge Device 149f
Group 44: [1022:149f] 20:07.0 Host bridge Device 149f
[1022:14a7] [R] 20:07.1 PCI bridge Device 14a7
[1022:14ac] [R] 93:00.0 Non-Essential Instrumentation [1300] Device 14ac
Group 45: [10b5:8724] [R] 21:00.0 PCI bridge PEX 8724 24-Lane, 6-Port PCI Express Gen 3 (8 GT/s) Switch, 19 x 19mm FCBGA
Group 46: [10b5:8724] [R] 22:00.0 PCI bridge PEX 8724 24-Lane, 6-Port PCI Express Gen 3 (8 GT/s) Switch, 19 x 19mm FCBGA
Group 47: [10b5:8724] 22:08.0 PCI bridge PEX 8724 24-Lane, 6-Port PCI Express Gen 3 (8 GT/s) Switch, 19 x 19mm FCBGA
Group 48: [10b5:8724] 22:09.0 PCI bridge PEX 8724 24-Lane, 6-Port PCI Express Gen 3 (8 GT/s) Switch, 19 x 19mm FCBGA
Group 49: [1000:0097] [R] 23:00.0 Serial Attached SCSI controller SAS3008 PCI-Express Fusion-MPT SAS-3
Group 50: [1000:0097] [R] 25:00.0 Serial Attached SCSI controller SAS3008 PCI-Express Fusion-MPT SAS-3
Group 51: [1b21:2421] [R] 26:00.0 PCI bridge Device 2421
Group 52: [1b21:2423] [R] 27:00.0 PCI bridge Device 2423
Group 53: [1b21:2423] [R] 27:01.0 PCI bridge Device 2423
[8086:15ef] [R] 5c:00.0 PCI bridge JHL7540 Thunderbolt 3 Bridge [Titan Ridge DD 2018]
[8086:15ef] [R] 5d:02.0 PCI bridge JHL7540 Thunderbolt 3 Bridge [Titan Ridge DD 2018]
[8086:15ef] 5d:04.0 PCI bridge JHL7540 Thunderbolt 3 Bridge [Titan Ridge DD 2018]
[8086:15f0] [R] 5e:00.0 USB controller JHL7540 Thunderbolt 3 USB Controller [Titan Ridge DD 2018]
Group 54: [1b21:2423] [R] 27:02.0 PCI bridge Device 2423
[1b21:2426] [R] 8c:00.0 USB controller Device 2426
Group 55: [1b21:2423] [R] 27:03.0 PCI bridge Device 2423
[1b21:2425] [R] 8d:00.0 USB controller Device 2425
Group 56: [144d:a80c] [R] 8e:00.0 Non-Volatile memory controller NVMe SSD Controller S4LV008[Pascal]
Group 57: [1022:43f4] [R] 8f:00.0 PCI bridge 600 Series Chipset PCIe Switch Upstream Port
Group 58: [1022:43f5] [R] 90:0c.0 PCI bridge 600 Series Chipset PCIe Switch Downstream Port
[1022:43f8] [R] 91:00.0 USB controller Device 43f8
USB: [046b:ff10] Bus 007 Device 009 American Megatrends, Inc. Virtual Keyboard and Mouse
USB: [046b:ffb0] Bus 007 Device 008 American Megatrends, Inc. Virtual Ethernet.
USB: [046b:ff31] Bus 007 Device 007 American Megatrends, Inc. Virtual HDisk Device
USB: [046b:ff20] Bus 007 Device 006 American Megatrends, Inc. Virtual CDROM
USB: [046b:ff01] Bus 007 Device 005 American Megatrends, Inc. Virtual Hub
USB: [1a86:55d4] Bus 007 Device 004 QinHeng Electronics SONOFF Zigbee 3.0 USB Dongle Plus V2
USB: [046d:c547] Bus 007 Device 003 Logitech, Inc. USB Receiver
USB: [046d:c539] Bus 007 Device 002 Logitech, Inc. Lightspeed Receiver
USB: [1d6b:0002] Bus 007 Device 001 Linux Foundation 2.0 root hub
USB: [1d6b:0003] Bus 008 Device 001 Linux Foundation 3.0 root hub
Group 59: [1022:43f5] 90:0d.0 PCI bridge 600 Series Chipset PCIe Switch Downstream Port
[1022:43f6] [R] 92:00.0 SATA controller 600 Series Chipset SATA Controller
 
Last edited:
mikeone said:
I have already checked and used also another similar script to have even more details, please see the output below.
Click to expand...

Yes, sorry, I misread your intial post and thought your host crashes when passing through the TB controller. Seems like there are some issues with power state management of your TB controller. You could try turning off PCIe power management via the kernel cmdline flag pcie_aspm=off
 
  • Like
Reactions: mikeone
I have created the cmdline file in the /etc/kernel directory and added the pcie_aspm=off line. also runned proxmox-boot-tool refresh and rebooted, but the issue is not solve I keep getting the same error messages.

Is there a way to know if the argument has been applied ?
 
I think I am not used system-boot but rather grub, only zfs is supported if I am not mistaken and I am using ext4, I'll do the same with grub and report back.
 
added to grub this is the result of cat /proc/cmdline.

Code: 
cat /proc/cmdline
BOOT_IMAGE=/boot/vmlinuz-6.8.4-3-pve root=/dev/mapper/pve-root ro quiet,pcie_aspm=off

Looks ok to me, however I am still getting errors, any idea on how to solve this ?

root@pve:~# dmesg | grep -i vfio
[ 12.745068] VFIO - User Level meta-driver version: 0.3
[ 22.035229] vfio-pci 0000:c1:00.0: vgaarb: deactivate vga console
[ 22.035235] vfio-pci 0000:c1:00.0: vgaarb: VGA decodes changed: olddecodes=io+mem,decodes=io+mem:eek:wns=none
[ 22.035799] vfio-pci 0000:c1:00.0: vgaarb: VGA decodes changed: olddecodes=io+mem,decodes=io+mem:eek:wns=none
[ 22.035844] vfio-pci 0000:c1:00.0: vgaarb: deactivate vga console
[ 22.035849] vfio-pci 0000:c1:00.0: vgaarb: VGA decodes changed: olddecodes=io+mem,decodes=io+mem:eek:wns=none
[ 178.563805] vfio-pci 0000:5e:00.0: Unable to change power state from unknown to D0, device inaccessible
[ 178.563816] vfio-pci 0000:5e:00.0: Unable to change power state from D3cold to D0, device inaccessible
[ 247.148109] vfio-pci 0000:5e:00.0: Unable to change power state from D3cold to D0, device inaccessible
[ 247.148115] vfio-pci 0000:5e:00.0: Unable to change power state from D3cold to D0, device inaccessible
[ 247.154669] vfio-pci 0000:5e:00.0: Unable to change power state from D3cold to D0, device inaccessible
[ 320.874508] vfio-pci 0000:5e:00.0: Unable to change power state from D3cold to D0, device inaccessible
[ 320.874515] vfio-pci 0000:5e:00.0: Unable to change power state from D3cold to D0, device inaccessible
[ 320.874550] vfio-pci 0000:5e:00.0: Unable to change power state from D3cold to D0, device inaccessible
[ 320.879210] vfio-pci 0000:5e:00.0: Unable to change power state from D3cold to D0, device inaccessible
 
The command-line still looks wrong to me, you have a comma instead of a space between quiet and pcie_aspm
 
indeed my bad, I have corrected it and now I am able to start the VM with the dock passed trough. thank you for your help !
I still need to figure out how to make it work with dp passtrough to the gpu
 
  • Like
Reactions: shanreich
I have tried everything but the dock only works for the display nothing I plug in to get detected. should I pass something else in addition to the "[8086:15f0] [R] 5e:00.0 USB controller JHL7540 Thunderbolt 3 USB Controller [Titan Ridge DD 2018]" ?

What options should I select when passing trough devices ? all functions - rom - Pci Express ?
 
I want to pass a thunderbolt controller through to a macOS VM and can’t get it recognized (It was recognized bare metal).

Have you had any luck yet?
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back