Limited bandwidth using Linux

[mikeone]

Hi all,

First of all let me details my specs to give a better context:

ASUS WRX90-E Sage
AMD TR Pro 7975WX
8X64 DDR5 5600Mhz
4x4To 990 Pro nvme ssd
2x RTX 3090
LSI 9300 HBA Card

In order to use my racked Workstation/GamingVM I am using a thunderbolt4 dock I am able to use the dock and add connected devices into the vm like mouse & keyboard but not passthrough the whole TB4 controller, but that is not a real concern however my main issue is to be able to use a long TB cable I am using a 50m Corning Optical cable but it doesn't work directly connected to the USB4 port of the motherboard I have to daisy chain first my belkin tb3 pro and my caldigit TS4.

This configuration is working perfectly in windows I am able to have my 3840x1600 monitor running at max refresh rate 175hz and connected my Mouse & keyboard headphones and speakers in addition to my external nvme ssd.

however when booting in proxmox with the same daisy chained configuration I am only able to get video output not a single device plugged to both dock works, seem to be an issue with XHCI please see dmseg log bellow.

Jul 23 20:15:48.466877 pve kernel: xhci_hcd 0000:29:00.0: Abort failed to stop command ring: -110
Jul 23 20:15:48.466883 pve kernel: fbcon: Taking over console
Jul 23 20:15:48.466974 pve kernel: xhci_hcd 0000:29:00.0: xHCI host controller not responding, assume dead
Jul 23 20:15:48.467064 pve kernel: xhci_hcd 0000:29:00.0: HC died; cleaning up
Jul 23 20:15:48.467069 pve kernel: clocksource: Long readout interval, skipping watchdog check: cs_nsec: 11966549563 wd_nsec: 11966554197
Jul 23 20:15:48.467075 pve kernel: usb usb8-port1: couldn't allocate usb_device
Jul 23 20:15:48.467165 pve kernel: xhci_hcd 0000:29:00.0: Error while assigning device slot ID: Command Aborted
Jul 23 20:15:48.467257 pve kernel: xhci_hcd 0000:29:00.0: Max number of devices this xHCI host supports is 64.
Jul 23 20:15:48.467263 pve kernel: usb usb5-port2: couldn't allocate usb_device
Jul 23 20:15:48.467269 pve kernel: usb usb6-port2: couldn't allocate usb_device

I also stumbled across PCIe bandwith issue scrolling in the dmseg logs.

root@pve:~# sudo dmesg | grep 'PCIe bandwidth'
[    1.426778] pci 0000:27:00.0: 2.000 Gb/s available PCIe bandwidth, limited by 2.5 GT/s PCIe x1 link at 0000:22:00.0 (capable of 31.504 Gb/s with 8.0 GT/s PCIe x4 link)
[    1.428768] pci 0000:29:00.0: 2.000 Gb/s available PCIe bandwidth, limited by 2.5 GT/s PCIe x1 link at 0000:22:00.0 (capable of 31.504 Gb/s with 8.0 GT/s PCIe x4 link)
[    1.429943] pci 0000:2a:00.0: 2.000 Gb/s available PCIe bandwidth, limited by 2.5 GT/s PCIe x1 link at 0000:22:00.0 (capable of 31.504 Gb/s with 8.0 GT/s PCIe x4 link)
[    1.437212] pci 0000:2c:00.0: 2.000 Gb/s available PCIe bandwidth, limited by 2.5 GT/s PCIe x1 link at 0000:22:00.0 (capable of 31.504 Gb/s with 8.0 GT/s PCIe x4 link)
[    1.438844] pci 0000:30:00.0: 2.000 Gb/s available PCIe bandwidth, limited by 2.5 GT/s PCIe x1 link at 0000:22:00.0 (capable of 4.000 Gb/s with 5.0 GT/s PCIe x1 link)
[    1.439579] pci 0000:5f:00.0: 2.000 Gb/s available PCIe bandwidth, limited by 2.5 GT/s PCIe x1 link at 0000:22:01.0 (capable of 15.753 Gb/s with 16.0 GT/s PCIe x1 link)
[    1.440684] pci 0000:61:00.0: 2.000 Gb/s available PCIe bandwidth, limited by 2.5 GT/s PCIe x1 link at 0000:22:01.0 (capable of 63.012 Gb/s with 16.0 GT/s PCIe x4 link)
[    2.989868] igc 0000:30:00.0: 2.000 Gb/s available PCIe bandwidth, limited by 2.5 GT/s PCIe x1 link at 0000:22:00.0 (capable of 4.000 Gb/s with 5.0 GT/s PCIe x1 link)

bandwidth limited message in dmesg, I don't know why, I should have plenty of bandwidth I am running a Threadripper pro 7975wx with an ASUS WRX90-E Sage.

Thank you for your help