Has this always been an issue, is this a new VM, or is this new since a Proxmox update? There are currently ongoing kernel issues in Proxmox for both PCI passthrough and USB (though I'm only aware of USB storage devices, but maybe it's all USB?).
But let's pretend this isn't a kernel issue...