USB Controller Passthrough = Proxmox Crash (Connection error)

[ZEMPA]

Hey everyone,

I'm new to the topic of Proxmox and have been encountering some issues with USB controller passthrough. I'm hoping to get some guidance on this issue.

System: Ryzen 5th gen with AMD 6600
After a lot of testing and reading many information about passthrough... less seems sometimes to be more.
(No edit to Grub, nothing in vifo, nothing in blacklist)
Currently working: PCI Passthrough: GPU, Audio

I would like to add the USB Controller to passthrough so that anything is just forwarded.
When I add the USB controller to the VM, the VM runs fine but Proxmox starts immediatly freezing and crashing and is completley unresposnsive. SSH also does not work anymore. So Everything times out, but the VM still runs without issues. I'm not sure why this is happening, and I've been unable to find a solution to this problem.

If anyone has any experience with this issue or has any suggestions on what I could try, please let me know. Also, please keep in mind that I am new to the topic of Proxmox, so any suggestions or guidance should be easy to understand.

I also played around and tried some stuff with vfio-pci for USB-Controller.
Also i tried several different kernels.

5.13.19-6-pve
5.15.85-1-pve [Problems with GPU Passthrough]
6.1.6-1-pve

Controller I want to passthrough

02:00.0 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Device [1022:43ee]
        Subsystem: ASMedia Technology Inc. Device [1b21:1142]
        Kernel driver in use: vfio-pci
        Kernel modules: xhci_pci

tried:

softdep xhci_pci pre: vfio-pci
options vfio-pci ids=1022:43ee

Thanks in advance for your help!

 

[leesteken]

When I add the USB controller to the VM, the VM runs fine but Proxmox starts immediatly freezing and crashing and is completley unresposnsive. SSH also does not work anymore. So Everything times out, but the VM still runs without issues. I'm not sure why this is happening, and I've been unable to find a solution to this problem.

Most likely, the USB controller is in the same IOMMU group as your network device and some other devices essential for Proxmox. You cannot share devices from the same IOMMU group between VMs or between a VM and the host.
The groups are determined by the motherboard (physical layout and BIOS) and tell you which device are, and which are not (same group), securly isolated from each other by IOMMU hardware. This is nothing new but hardly mentioned in documentation and guides. Can you tell us the make and model of your motherboard?

Please show your IOMU groups with this command: for d in /sys/kernel/iommu_groups/*/devices/*; do n=${d#*/iommu_groups/*}; n=${n%%/*}; printf 'IOMMU group %s ' "$n"; lspci -nns "${d##*/}"; done. Passthrough a different USB controller or buy a separate USB controller PCIe card (which is know to work well with passthrough) and hope it ends up in a separate group.

 

[ZEMPA]

Ah thanks and you seem to be right. Its Group 15 with LAN Controller included. So this means I only can add everything from the IOMMU Group and can not specify (subgroup) it any more, right? - I understand a sperate USB Controller would be the best but than I would rather route every USB Port to the VM.

Can you tell us the make and model of your motherboard?

Asus TUF GAMING B550-PLUS

Please show your IOMU groups with this command: for d in /sys/kernel/iommu_groups/*/devices/*; do n=${d#*/iommu_groups/*}; n=${n%%/*}; printf 'IOMMU group %s ' "$n"; lspci -nns "${d##*/}"; done. Passthrough a different USB controller or buy a separate USB controller PCIe card (which is know to work well with passthrough) and hope it ends up in a separate group.

root@pve:~# for d in /sys/kernel/iommu_groups/*/devices/*; do n=${d#*/iommu_groups/*}; n=${n%%/*}; printf 'IOMMU group %s ' "$n"; lspci -nns "${d##*/}"; done
IOMMU group 0 00:01.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU group 10 00:08.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU group 11 00:08.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] [1022:1484]
IOMMU group 12 00:14.0 SMBus [0c05]: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller [1022:790b] (rev 61)
IOMMU group 12 00:14.3 ISA bridge [0601]: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge [1022:790e] (rev 51)
IOMMU group 13 00:18.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 0 [1022:1440]
IOMMU group 13 00:18.1 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 1 [1022:1441]
IOMMU group 13 00:18.2 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 2 [1022:1442]
IOMMU group 13 00:18.3 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 3 [1022:1443]
IOMMU group 13 00:18.4 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 4 [1022:1444]
IOMMU group 13 00:18.5 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 5 [1022:1445]
IOMMU group 13 00:18.6 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 6 [1022:1446]
IOMMU group 13 00:18.7 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 7 [1022:1447]
IOMMU group 14 01:00.0 Non-Volatile memory controller [0108]: Sandisk Corp Device [15b7:5030] (rev 01)
IOMMU group 15 02:00.0 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Device [1022:43ee]
IOMMU group 15 02:00.1 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] Device [1022:43eb]
IOMMU group 15 02:00.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43e9]
IOMMU group 15 03:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43ea]
IOMMU group 15 03:08.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43ea]
IOMMU group 15 03:09.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43ea]
IOMMU group 15 06:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8125 2.5GbE Controller [10ec:8125] (rev 05)
IOMMU group 16 07:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 XL Upstream Port of PCI Express Switch [1002:1478] (rev c7)
IOMMU group 17 08:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 XL Downstream Port of PCI Express Switch [1002:1479]
IOMMU group 18 09:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 23 [1002:73ff] (rev c7)
IOMMU group 19 09:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:ab28]
IOMMU group 1 00:01.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge [1022:1483]
IOMMU group 20 0a:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Function [1022:148a]
IOMMU group 21 0b:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP [1022:1485]
IOMMU group 22 0b:00.1 Encryption controller [1080]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Cryptographic Coprocessor PSPCPP [1022:1486]
IOMMU group 23 0b:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller [1022:149c]
IOMMU group 24 0b:00.4 Audio device [0403]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse HD Audio Controller [1022:1487]
IOMMU group 2 00:01.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge [1022:1483]
IOMMU group 3 00:02.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU group 4 00:03.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU group 5 00:03.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge [1022:1483]
IOMMU group 6 00:04.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU group 7 00:05.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU group 8 00:07.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU group 9 00:07.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] [1022:1484]

 

[leesteken]

Ah thanks and you seem to be right. Its Group 15 with LAN Controller included. So this means I only can add everything from the IOMMU Group and can not specify (subgroup) it any more, right? - I understand a sperate USB Controller would be the best but than I would rather route every USB Port to the VM.

Asus TUF GAMING B550-PLUS

Every Ryzen motherboard except X570 has a big "chipset" IOMMU group and has the same issues. A separate USB controller PCIe card won't help because it will probably also end up in that big group (unless you put it in the x16 slot of the GPU but then the GPU passthrough will give you trouble).

IOMMU group 23 0b:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller [1022:149c]

This USB controller probably comes from the Ryzen CPU and is in a separate IOMMU group. You should not have much trouble passing it through to your VM.

 

[ZEMPA]

Yea I have seen and added the other USB Controller but none of the devices are picked up. I dont want to say its not working but all ports I am using seem connected to 00:02:00.

Thanks for saving my sanity and much time doing more try and error. So I will forward the ports.

 

[leesteken]

Yea I have seen and added the other USB Controller but none of the devices are picked up. I dont want to say its not working but all ports I am using seem connected to 00:02:00.

That's uncommon. Maybe some ports (like on the from of the PC case) can be plugged into another place on the motherboard?

Thanks for saving my sanity and much time doing more try and error. So I will forward the ports.

That never gives good results for low-latency or high-bandwidth devices and I have even experienced dropped key presses, but maybe it works well enough for you.

 

[ZEMPA]

Thanks for you help and advice
I will do some more testing and will check out if I can rearrange some ports.

 

[branpurn]

I am experiencing this exact same issue. Please do share anything you find.

I have an ASMedia ASM3242 controller on a separate PCIe card that I can passthrough to the virtual machine, but when I do, all devices in that IOMMU group are removed from the hypervisor, which is virtually everything that has no relation to the ASM3242.

 

[ZEMPA]

I checked the speed of the USB Ports. And using it with a moderate USB SSD it got more than 200MB/s which seems enough an fine for me. So the current state should not be an issue for me.
If I have more info I will post them here. But as of today I am fine with the current state.

 

[branpurn]

In my case, the controller I am trying to pass through is 20 Gbps USB-C connection, so any way to overcome the entire IOMMU group going along for the ride is a big deal.

 

[leesteken]

In my case, the controller I am trying to pass through is 20 Gbps USB-C connection, so any way to overcome the entire IOMMU group going along for the ride is a big deal.

Splitting devices of the same group over different VMs and/or the Proxmox host is insecure. The VM can read the other VMs or the Proxmox host memory via DMA initiated by the PCI(e) devices. Just in case you don't care about security or VM insolation, search this forum for the built-in pcie_acs_override kernel parameter.

 

[mr44er]

If I have more info I will post them here. But as of today I am fine with the current state.

Just an idea, could it be that IOMMU group 15 02:00.0 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Device [1022:43ee] is wired to the ports on the back and IOMMU group 23 0b:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller [1022:149c] is the internal port?
https://dlcdnets.asus.com/pub/ASUS/mb/SocketAM4/TUF_GAMING_B550-PLUS/E16576_TUF_GAMING_B550-PLUS_UM_WEB.pdf?model=TUF GAMING B550-PLUS <- Page 11/34 Point 8

On my X570 are the ports on the back wired as shown. Maybe it's the same for you?

 

[batat]

I've been fighting with my macbook pro 2013 with a similar issue. I'm using it as the proxmox host and I'm trying to pass through a USB Hub to a VM so I can pas through external drives too. I also have a USB dongle for ethernet connection, and whenever I select the xHCI controller that has the "Mass Storage" and it's the seagate drive, and start the VM, proxmox host loses internet connection. I couldn't figure out why, until it clicked to me that it's passing the entire Bus04 to the VM, and that means both USB ports of the macbook, the ethernet usb dongle and the usb hub. Although the lsusb -toutput shows two Buses with Driver=xhci_hcd and was hoping that the ethernet usb dongle is on a separate bus, but it doesn't seem to be that way.

lsusb -t

/:  Bus 04.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/4p, 5000M
    |__ Port 1: Dev 3, If 0, Class=Vendor Specific Class, Driver=ax88179_178a, 5000M
    |__ Port 2: Dev 2, If 0, Class=Hub, Driver=hub/4p, 5000M
        |__ Port 1: Dev 4, If 0, Class=Mass Storage, Driver=usbfs, 5000M
/:  Bus 03.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/4p, 480M
    |__ Port 2: Dev 3, If 0, Class=Hub, Driver=hub/4p, 480M
/:  Bus 02.Port 1: Dev 1, Class=root_hub, Driver=ehci-pci/2p, 480M
    |__ Port 1: Dev 2, If 0, Class=Hub, Driver=hub/8p, 480M
        |__ Port 8: Dev 3, If 0, Class=Hub, Driver=hub/2p, 480M
            |__ Port 1: Dev 4, If 0, Class=Hub, Driver=hub/3p, 12M
                |__ Port 3: Dev 8, If 1, Class=Wireless, Driver=btusb, 12M
                |__ Port 3: Dev 8, If 2, Class=Vendor Specific Class, Driver=btusb, 12M
                |__ Port 3: Dev 8, If 0, Class=Vendor Specific Class, Driver=btusb, 12M
                |__ Port 3: Dev 8, If 3, Class=Application Specific Interface, Driver=, 12M
            |__ Port 2: Dev 5, If 0, Class=Human Interface Device, Driver=usbhid, 12M
            |__ Port 2: Dev 5, If 1, Class=Human Interface Device, Driver=usbhid, 12M
            |__ Port 2: Dev 5, If 2, Class=Human Interface Device, Driver=usbmouse, 12M
/:  Bus 01.Port 1: Dev 1, Class=root_hub, Driver=ehci-pci/2p, 480M
    |__ Port 1: Dev 2, If 0, Class=Hub, Driver=hub/6p, 480M
        |__ Port 1: Dev 3, If 0, Class=Video, Driver=uvcvideo, 480M
        |__ Port 1: Dev 3, If 1, Class=Video, Driver=uvcvideo, 480M
        |__ Port 1: Dev 3, If 2, Class=Vendor Specific Class, Driver=, 480M

Bus04.Port1 > Port 2 > Port 1 is my external drvie. I'm assuming its parent Port 2 is the USB hub, and I just looked up Driver=ax88179 and it is the ethernet USB dongle.

I opted for using the USB Device hardware option for the VM instead PCI passthrough. I will see how that goes, I might end up having to use that drive as a NFS for flexibility, but we'll see.

I thought I would add this here in case someone in the future wondering why they can't pass through their external drive on macbook to a vm while using a usb dongle ethernet. And to see if there is a way to select to pass only Port2 instead of the entire Bus04.
For now, the USB device option will be enough, unless I notice a really bad performance, but I doubt I will because it's just an external drive to be a backup for a backup and to test the reliability of this setup.