Hi,
Take a look here:
https://forum.proxmox.com/threads/vm-freezes-when-disconnecting-usb-keyboard-and-mouse.70287/post-341950
Updating the pve-qemu-kvm package to a version greater than 5.1.0-1 seems to resolve the issue.
I can confirm that updating the pve-qemu-kvm package seems to have resolved the problem for me! :)
Since updating yesterday, I was able to remove several devices multiple times which are passed through to two VMs without any locking up. Before there was about a 50% chance of the VM locking.