USB PCIe device dead on boot - xHCI host controller not responding, assume dead

[spookky]

Hello all, I'm having some trouble getting a PCIe usb controller working with my system and I'm not sure where the exact problem is. As an FYI, I am successfully passing 3 GPUs (1660 Ti, 3060 Ti, and a RX 6400) and 2 usb controllers (they are motherboard controllers and NOT pcie add-ons) as well as 3 NVMe 1TB samsung drives to 3 seperate vms. No troubles with anything other than this pcie usb controller card. I have tried 3 usb controllers now and experience similar issues from all. They all appear to be asmedia chips though... although some of the board mounted chips are ALSO asmedia chips and I have no issue with those.

The motherboard is Supermicro M12SWA-TF with AMD Ryzen Threadripper Pro 3955WX

It appears on surface very similar to the following issue raised here.

https://forum.proxmox.com/threads/pcie-passthrough-not-working.113938/

The controller(s) and IOMMU group I'm trying to pass through is...

46:00.0 USB controller [0c03]: ASMedia Technology Inc. ASM2142 USB 3.1 Host Controller [1b21:2142] (prog-if 30 [XHCI])
        Subsystem: Sonnet Technologies, Inc. ASM2142 USB 3.1 Host Controller [16b8:7337]
        Flags: bus master, fast devsel, latency 0, IRQ 159, IOMMU group 43
        Memory at c9100000 (64-bit, non-prefetchable) [size=32K]
        Capabilities: [50] MSI: Enable- Count=1/8 Maskable- 64bit+
        Capabilities: [68] MSI-X: Enable+ Count=8 Masked-
        Capabilities: [78] Power Management version 3
        Capabilities: [80] Express Legacy Endpoint, MSI 00
        Capabilities: [100] Advanced Error Reporting
        Capabilities: [200] Secondary PCI Express
        Capabilities: [300] Latency Tolerance Reporting
        Kernel driver in use: xhci_hcd
        Kernel modules: xhci_pci

47:00.0 USB controller [0c03]: ASMedia Technology Inc. ASM2142 USB 3.1 Host Controller [1b21:2142] (prog-if 30 [XHCI])
        Subsystem: Sonnet Technologies, Inc. ASM2142 USB 3.1 Host Controller [16b8:7337]
        Flags: bus master, fast devsel, latency 0, IRQ 168, IOMMU group 44
        Memory at c9000000 (64-bit, non-prefetchable) [size=32K]
        Capabilities: [50] MSI: Enable- Count=1/8 Maskable- 64bit+
        Capabilities: [68] MSI-X: Enable+ Count=8 Masked-
        Capabilities: [78] Power Management version 3
        Capabilities: [80] Express Legacy Endpoint, MSI 00
        Capabilities: [100] Advanced Error Reporting
        Capabilities: [200] Secondary PCI Express
        Capabilities: [300] Latency Tolerance Reporting
        Kernel driver in use: xhci_hcd
        Kernel modules: xhci_pci

/etc/modprobe.d/vfio.conf

options vfio-pci ids=10de:2182,10de:1aeb,10de:1aec,10de:1aed,1b21:3242,1002:743f,1002:ab28
options vfio-pci ids=1b21:2142,16b8:7337

That is all apart of 1 pcie card...

https://www.amazon.com/dp/B07LG5TS6H?psc=1&ref=ppx_yo2ov_dt_b_product_details

While I am able to pass this through and see it in a Windows 10 VM and have issues like the forum post, I have noticed an issue during boot found in dmesg. During boot I see the following associated with device (I have a usb logitech receiver for keyboard and mouse plugged into a port associated with this controller) connected to 47:00

root@mandv:~# dmesg | grep 47:00
[    1.915869] pci 0000:47:00.0: [1b21:2142] type 00 class 0x0c0330
[    1.915894] pci 0000:47:00.0: reg 0x10: [mem 0xc9000000-0xc9007fff 64bit]
[    1.915953] pci 0000:47:00.0: enabling Extended Tags
[    1.916032] pci 0000:47:00.0: PME# supported from D0
[    1.916083] pci 0000:47:00.0: 8.000 Gb/s available PCIe bandwidth, limited by 5.0 GT/s PCIe x2 link at 0000:45:02.0 (capable of 15.752 Gb/s with 8.0 GT/s PCIe x2 link)
[    1.953614] pci 0000:47:00.0: PME# does not work under D0, disabling it
[    1.955661] pci 0000:47:00.0: Adding to iommu group 44
[    3.240816] xhci_hcd 0000:47:00.0: xHCI Host Controller
[    3.240818] xhci_hcd 0000:47:00.0: new USB bus registered, assigned bus number 9
[    3.295647] xhci_hcd 0000:47:00.0: hcc params 0x0200ef80 hci version 0x110 quirks 0x0000000000800010
[    3.295972] xhci_hcd 0000:47:00.0: xHCI Host Controller
[    3.295974] xhci_hcd 0000:47:00.0: new USB bus registered, assigned bus number 10
[    3.295975] xhci_hcd 0000:47:00.0: Host supports USB 3.1 Enhanced SuperSpeed
[    3.296024] usb usb9: SerialNumber: 0000:47:00.0
[    3.296148] usb usb10: SerialNumber: 0000:47:00.0
[   13.999126] xhci_hcd 0000:47:00.0: Abort failed to stop command ring: -110
[   13.999183] xhci_hcd 0000:47:00.0: xHCI host controller not responding, assume dead
[   13.999198] xhci_hcd 0000:47:00.0: HC died; cleaning up
[   13.999225] xhci_hcd 0000:47:00.0: Timeout while waiting for setup device command

I've tried everything I can find concerning kernel parameters (usb suspend, unsafe interrupts, etc.) bios configurations (ensuring IOMMU is enabled, disabling OPROM on the pcie slot, etc.) i've tried all manner of vm configurations; however, I'm feeling pretty confident that until I can get the base OS to work with the pcie card I won't be able to get the VM to work.

Even though I have early binding (which I have also tried inserting as a kernel parameter) the usb controllers ALWAYS show up with the xhci_hcd driver in use. I have tried keeping the usb device disconnected during start-up and unbinding and binding to vfio (either by just starting the VM the controller is attached to or by bash commands) and even though the controller will appear in the VM I am unable to use any usb device attached to it.

I'm at my wits' end and appreciate any and all help you all may be able to offer. Please let me know if I can provide more information, I'd really love to get this one figured out.

 

[leesteken]

Even though I have early binding (which I have also tried inserting as a kernel parameter) the usb controllers ALWAYS show up with the xhci_hcd driver in use. I have tried keeping the usb device disconnected during start-up and unbinding and binding to vfio (either by just starting the VM the controller is attached to or by bash commands) and even though the controller will appear in the VM I am unable to use any usb device attached to it.

Have you tried adding softdep xhci_pci pre: vfio_pci to make sure vfio-pci is loaded before the actual driver? That works for my ASMedia device.

 

[spookky]

No dice...

cat /etc/modprobe.d/vfio.conf
options vfio-pci ids=10de:2182,10de:1aeb,10de:1aec,10de:1aed,1b21:3242,1002:743f,1002:ab28
options vfio-pci ids=1b21:2142,16b8:7337
softdep xhci_hcd pre: vfio-pci

dmesg showing xhci driver loaded first... vfio driver and then failure. Interestingly enough, despite having early binding specified for many PCIe devices the only one that comes up is the usb controller (I assume I'm interpreting this correctly?)

dmesg | grep -e 0000:47:00 -e 0000:46:00 -e 1b21 -e vfio
[    1.909995] pci 0000:66:00.0: [1b21:3242] type 00 class 0x0c0330
[    1.916228] pci 0000:46:00.0: [1b21:2142] type 00 class 0x0c0330
[    1.916253] pci 0000:46:00.0: reg 0x10: [mem 0xc9100000-0xc9107fff 64bit]
[    1.916312] pci 0000:46:00.0: enabling Extended Tags
[    1.916390] pci 0000:46:00.0: PME# supported from D0
[    1.916441] pci 0000:46:00.0: 8.000 Gb/s available PCIe bandwidth, limited by 5.0 GT/s PCIe x2 link at 0000:45:01.0 (capable of 15.752 Gb/s with 8.0 GT/s PCIe x2 link)
[    1.916532] pci 0000:47:00.0: [1b21:2142] type 00 class 0x0c0330
[    1.916557] pci 0000:47:00.0: reg 0x10: [mem 0xc9000000-0xc9007fff 64bit]
[    1.916615] pci 0000:47:00.0: enabling Extended Tags
[    1.916693] pci 0000:47:00.0: PME# supported from D0
[    1.916745] pci 0000:47:00.0: 8.000 Gb/s available PCIe bandwidth, limited by 5.0 GT/s PCIe x2 link at 0000:45:02.0 (capable of 15.752 Gb/s with 8.0 GT/s PCIe x2 link)
[    1.954060] pci 0000:46:00.0: PME# does not work under D0, disabling it
[    1.954184] pci 0000:47:00.0: PME# does not work under D0, disabling it
[    1.956141] pci 0000:46:00.0: Adding to iommu group 43
[    1.956161] pci 0000:47:00.0: Adding to iommu group 44
[    3.102671] xhci_hcd 0000:46:00.0: xHCI Host Controller
[    3.102674] xhci_hcd 0000:46:00.0: new USB bus registered, assigned bus number 7
[    3.157520] xhci_hcd 0000:46:00.0: hcc params 0x0200ef80 hci version 0x110 quirks 0x0000000000800010
[    3.157905] xhci_hcd 0000:46:00.0: xHCI Host Controller
[    3.157908] xhci_hcd 0000:46:00.0: new USB bus registered, assigned bus number 8
[    3.157911] xhci_hcd 0000:46:00.0: Host supports USB 3.1 Enhanced SuperSpeed
[    3.157964] usb usb7: SerialNumber: 0000:46:00.0
[    3.158108] usb usb8: SerialNumber: 0000:46:00.0
[    3.158285] xhci_hcd 0000:47:00.0: xHCI Host Controller
[    3.158288] xhci_hcd 0000:47:00.0: new USB bus registered, assigned bus number 9
[    3.213131] xhci_hcd 0000:47:00.0: hcc params 0x0200ef80 hci version 0x110 quirks 0x0000000000800010
[    3.213476] xhci_hcd 0000:47:00.0: xHCI Host Controller
[    3.213478] xhci_hcd 0000:47:00.0: new USB bus registered, assigned bus number 10
[    3.213480] xhci_hcd 0000:47:00.0: Host supports USB 3.1 Enhanced SuperSpeed
[    3.213529] usb usb9: SerialNumber: 0000:47:00.0
[    3.213654] usb usb10: SerialNumber: 0000:47:00.0
[    7.805126] vfio_pci: add [1b21:2142[ffffffff:ffffffff]] class 0x000000/00000000
[    7.805134] vfio_pci: add [16b8:7337[ffffffff:ffffffff]] class 0x000000/00000000
[   19.119654] xhci_hcd 0000:47:00.0: Abort failed to stop command ring: -110
[   19.119710] xhci_hcd 0000:47:00.0: xHCI host controller not responding, assume dead
[   19.119725] xhci_hcd 0000:47:00.0: HC died; cleaning up
[   19.119746] xhci_hcd 0000:47:00.0: Timeout while waiting for setup device command

Just as an FYI id 66:00 is the onboard asmedia 3142 chip.

 

[leesteken]

No dice...

cat /etc/modprobe.d/vfio.conf
options vfio-pci ids=10de:2182,10de:1aeb,10de:1aec,10de:1aed,1b21:3242,1002:743f,1002:ab28
options vfio-pci ids=1b21:2142,16b8:7337
softdep xhci_hcd pre: vfio-pci

Have you also tried softdep xhci_pci pre: vfio_pci? The xhci_pci was intentional: even though it shows xhci_hcd, the kernel driver is xhci_pci.
Are you sure you can split the options vfio-pci ids=... over two lines? I only use one with all IDs.

EDIT: It is quite possible that this won't help and you need a different (type or brand) USB controller. People often report that the USB controllers don't reset properly with passthrough, so it appears that you need to be lucky.

 

[spookky]

Ah got it, sorry about that. I did try _pci at first but let me try and clean up vfio.conf and give it another shot. I'm also trying to blacklist the xhci driver from even loading in the first place with a kernel parameter but it's likely not working because I specified it incorrectly. Need to try these things once at a time because I think I'm starting to mix and match and it will eventually cause other issues down the road, at least that's my assumption.

Thanks for your help!

 

[spookky]

Ok, blakclisting at least worked and now showing vfio as loaded first... need to unwind myself and figure out if this can pass through successfully to the VM and what bare minimum config is.

Also, you are correct, my multi-line ids was not the correct implementation

 

[spookky]

Not working.

VM boots fine showing the connected usb controller but as soon as a plug in a device I get code 43 on the controller

Windows has stopped this device because it has reported problems. (Code 43)

I've got a Reneas controller coming on Monday, guessing this one is very unlikely to work.

 

[spookky]

Closing the loop on this...

I purchased the following PCIe card and everything works... first try. Really wanted that dual controller to work for me but oh well, at least this one does.

https://www.amazon.com/dp/B01HR0PL0Y?psc=1&ref=ppx_yo2ov_dt_b_product_details

Thanks for the help!