USB 3 and Max number of devices this xHCI host supports is 32

[slickdakine]

Would like to pick some of your brains....I'm using a Startech PEXUSB3S44V USB 3 card:
https://www.startech.com/en-us/cards-adapters/pexusb3s44v
It has 4 root hubs:
lspci:
18:00.0 USB controller: Renesas Technology Corp. uPD720202 USB 3.0 Host Controller (rev 02)
19:00.0 USB controller: Renesas Technology Corp. uPD720202 USB 3.0 Host Controller (rev 02)
1a:00.0 USB controller: Renesas Technology Corp. uPD720202 USB 3.0 Host Controller (rev 02)
1b:00.0 USB controller: Renesas Technology Corp. uPD720202 USB 3.0 Host Controller (rev 02)

I'm running:
CPU: 32 x AMD Ryzen 9 5950X 16-Core Processor (1 Socket)
Kernel Version Linux 5.15.2-edge #1 SMP PVE Edge 5.15.2-1 generic
PVE: 7.1-5

I would like to plug in 15 hard drives to one of those root hubs. Each drive has a 2 port hub on it. They are plugged in a chain like follows using USB 3 cables:
1
11
1111
11111111

In Proxmox, I can use a USB 2.0 cable between the top of the chain and the card, and Proxmox will detect all 15 drives at USB 2.0 speeds.
This holds true in all versions of linux, including Fedora, Debian, Ubuntu, xcp-ng (Xen). If I use a USB 3.0 cable I get the following error in dmesg:
xhci_hcd 0000:0e:00.0: Max number of devices this xHCI host supports is 32.
And only 7 of the 15 drives are detected. This again holds true in all the above flavors of linux.

Now for the fun part.
Vmware 7 and FreeBSD both detect all 15 drives at 3.0 speeds with a 3.0 cable. Vmware allows me to pass all 15 drives to a VM which I can then mount and access the data. FreeBSD 13 (latest) also allows me to mount/access all 15 drives.

I've done a lot of reading on the subject, and many point out this is a hardware limitation:
http://marc.merlins.org/perso/linux...f-devices-this-xHCI-host-supports-is-32_.html
https://acroname.com/blog/how-many-usb-devices-can-i-connect

If it is a hardware limitation, why does it work in Vmware and FreeBSD? From what I'm finding out, it seems to be a limitation with the Linux kernel.

I've also looked into doing a PCI passthrough etc...but wasn't able to get it to function correctly. From what I've read the startech card doesn't PCI passthrough well.
I also looked into a USB passthrough, but from from my attempts Proxmox needs to detect the device to pass it through. The specific driver that seems to be causing the problem is the xhci-pci/xhci_hcd driver.

I prefer to use Proxmox in my application. But for now I am stuck with Vmware and possibly FreeBSD. Any ideas?

 

[slickdakine]

Wanted to follow up. I was able to get it all to work well with VMWare. I can pass up to 20 drives per VMware linux guest. VMware detects all 15 drives.

 

[Tekkness]

Hey, I dont know if you are still having this issue, but for anyone else who is experiencing something similar:

In my case, I wanted to virtualize a Windows Server with had a function of SmartCard reader. I must admit I am not that knowledgeable with USB and Buses. All I know is that this smartcard reader has two USB connectors, which in turn are 20 USB resources, one for each card slot.

However, not all of them would work properly. The Debian 11 under Proxmox wouldnt read all of them, at most 11 cards would work. In "/var/log/kern.log", on the last few devices I would see the error "Not enough host controller resources for new device state", and "can't set config #1, error -12"

I found this post: https://itectec.com/superuser/linux-not-enough-host-controller-resources-for-new-device-state/

Disabling in the BIOS "xHCI" works, and now all the USB devices are read properly, and the USB passthrough for all 20 USB card works perfectly. It does mean however, that your USB3 devices will only work as USB2. I dont need USB3 in this case, so it doesnt matter to me