I had a similar issue with a usb modem a few years back, in the end I resorted to passing through a whole usb controller via vfio passthrough, luckily the board had 2 separate iommu groups for the controllers, so the host kept one and guest got the other, problem was solved in that particular...