PCI passthrough only works after a VM shutdown + restart

[promoxer]

Group 16:       [10de:1382] [R] 08:00.0  VGA compatible controller                GM107 [GeForce GTX 745]
                [10de:0fbc]     08:00.1  Audio device                             GM107 High Definition Audio Controller [GeForce 940MX]
Group 17:       [1022:148a] [R] 09:00.0  Non-Essential Instrumentation [1300]     Starship/Matisse PCIe Dummy Function
Group 18:       [1022:1485] [R] 0a:00.0  Non-Essential Instrumentation [1300]     Starship/Matisse Reserved SPP
Group 19:       [1022:1486] [R] 0a:00.1  Encryption controller                    Starship/Matisse Cryptographic Coprocessor PSPCPP
Group 20:       [1022:149c] [R] 0a:00.3  USB controller                           Matisse USB 3.0 Host Controller
Group 21:       [1022:1487]     0a:00.4  Audio device                             Starship/Matisse HD Audio Controller

1. I'm passing through my discrete GPU 08:00.0 and onboard USB controller 0a:00.3 as 2 PCI devices to a Windows 10 VM
2. They are both in their own IOMMU groups
3. The GPU works without issues, however, the USB controller always fails on the 1st VM boot
4. It fails with an X beside the USB controller under Windows' Device Manager
5. 1st VM boot means the 1st time the VM is booted after a proxmox reboot
6. Thus, I have to shutdown the VM and boot it the 2nd time before everything works

Anyone ever experienced anything similar?