Can you repeat the behaviour in a linux guest? It normally gives better explanation on what goes wrong. Another setup would be starting MacOS up in debug mode, adding serial console and stuff like that, but even if you find the culprit, you may not be able to fix it.
I also tried this last year or the year before (time flyes ...) and I ended up buying different usb3 controllers for passthrough and the most expensive one worked (roughly 40 eur) fine in MacOS, Linux and Windows, so a fully virtualized machine with dedicated GPU and USB was possible.