The trick it seems is to manually add the usb devices. For eg:
args: -device nec-usb-xhci,id=xhci -device usb-host,bus=xhci.0,hostbus=4,hostport=2
Omitting bus and addr seems to allow the nec-usb-xhci driver to assign to pcie.
That said, my USB devices still don't like this and I'm getting...