[SOLVED] Odd USB Passthrough Issue

[Astraea]

I have just built a new Tower computer that has the following major components:

NVIDIA 1060 - Assigned to a Linux VM
NVIDIA 3080ti - Assigned to a Windows VM
5950x CPU
128GB RAM
2 x 500GB SSDs for Proxmox in a ZFS mirror
2 x 500GB NVMe disks for VM storage in a ZFS mirror


Everything is working really well except for 2 USB devices, when I connect my PS5 controller and my Razer Kiao camera I can only get one of them to work at a time but not both at the same time. It does not matter which port I use and I have tried passing through the device and also the port itself with no effect, I have tried to use one of the onboard USB Ports, Ports on a USB hub and also a front case port to see if it has to do with needing them on a separate USB controller.

I am hoping the Proxmox community has some solutions and or additional troubleshooting ideas to get this last little hiccup resolved.

 

[Stefan_R]

USB passthrough has very limited bandwidth and performance in general. Getting a webcam working at all is probably pushing the limits.

I'd recommend using PCIe passthrough to assign an entire USB controller directly to your VM. This will give you native performance and should work with any devices. Often times even the onboard controllers are in seperate IOMMU groups and can be used for this purpose.

 

[Astraea]

I havent tried passing through a controller but it is on my list to try. I did get both working by passing through the Bluetooth that is built into the motherboard though. This allowed for the controller to be connect via Bluetooth and the camera on USB. Interestingly the Bluetooth adapter was detected as a USB device.

Would I determine which controller a port is on using the same methods as finding out what controller an HDD/SSD is on?

Also is there a way to increase the limit on USB devices for VM? Should I create a bug/feature request to have it looked into to see if that can be added under the global settings or maybe a per node setting in the WebUI?

 

[Stefan_R]

lsusb -t is useful for mapping ports/controllers, but tbh I've always just used trial and error, usually the ports on a mainboard that share a controller are physically grouped together too.

Also is there a way to increase the limit on USB devices for VM? Should I create a bug/feature request to have it looked into to see if that can be added under the global settings or maybe a per node setting in the WebUI?

It is currently hardcoded into our stack. But as stated before, USB passthrough suffers from rather heavy latency/throughput penaltys (since it has to pass through QEMU in user space), so adding even more devices (and thus load) is very rarely the correct solution.

 

[Astraea]

I tried to pass through the controller to the VM but the rear ports are on separate controllers within the same group. When I pass through each controller to the VM I want only one machine will start. I have another controller in a different group but that is for the front ports.

Is there a way to attach the controllers that are part of the same group to 2 different VMs or is it an all or nothing?

I have also tried the pcie_acs_override=downstream option with no changes on being able to separate that group.

 

[Stefan_R]

It's all or nothing, the actual seperation depends on the motherboard hardware.

pcie_acs_override usually helps, but there are scenarios where it doesn't. Are you sure it's active? Check your kernel commandline with 'cat /proc/cmdline' to see if it applied correctly.

 

[Astraea]

So I enabled the override using the Systemd-boot method and was showing it was added with your command. When I look to assign the controllers they are still showing in the same group and when I go to start the second VM it fails saying the device is in use?

Is there another command that needs to be run in order to update the IOMMU groups?

 

[Astraea]

So I spent some more time tonight with this and while I got the override to work and did get the controller split. I started to see the following errors when the devices were split between 2 VMs.

kvm: vfio: unable to power on device, stuck in d3
kvm: vfio: Cannot reset device 0000:09:00.1, depends on group 31 which is not owned.

Though it did work when all three parts of the controller were on 1 VM. It would run when they were separate but there was no power to the USB ports and no devices would work.

I am now looking at mounting the 3080ti card vertically so I can access more of the PCI-e slots to add 2 additional USB cards for pass through.

---

So I am able to get 1 machine to work even with those errors but not the other, seems as though the first one to start wins the control of all of the ports, and was defiantly not stable. Definitely planning to look into using add-in cards vs using onboard devices for passthrough.

 

[Astraea]

I wanted to post a conclusion to this thread for others that come along later. While you can pass up to 5 USB devices to a VM it really only works well with low speed or low I/O devices, think keyboards, mice and older USB drives. Once you try to pass through I/O intensive devices you might only get one maybe two functioning, but you eat up the bandwidth that this process uses to get them to the VM very quickly.

I ended up getting a USB PCI-e 3.0 card and then assigning that to the Windows VM, this way I could use my PS5 controller, Camera and still have room for a 4 port 3.0 USB hub and a spare port for an external HDD or something else in the future. The Linux VM is mostly a workstation so it only needed a mouse, keyboard and a few ports for things like an external HDD or flash drive so I did the USB passthrough for 3 of the 4 ports on another USB hub I connected to the system.

I would have gone with another dedicated USB card but I ran out of PCI-e lanes and so it was either the USB card or the Network card and I would rather have a dedicated NIC for each VM vs faster USB ports and a USB to Ethernet adapter.

 

[leesteken]

I ended up getting a USB PCI-e 3.0 card and then assigning that to the Windows VM, this way I could use my PS5 controller, Camera and still have room for a 4 port 3.0 USB hub and a spare port for an external HDD or something else in the future.

Thank you for making the effort to inform future readers. As it turns out, a lot of USB PCIe cards don't work well with PCI passthrough or function level resets. If it's not too much trouble, could you share the brand and type of your card that works with Proxmox?

 

[Astraea]

Sure thing, I got it from Memory Express as I am in Canada but here are the details from the manufacturer.

StarTech - 4 Port PCI Express PCIe SuperSpeed USB 3.0 Controller Card Adapter with UASP - SATA Power - Part Number: PEXUSB3S4V

Here is a link to the product on the StarTech site: Click Here