I am also seeing this issue, and my setup is very similar with a USB 2.0 KVM switch, but unfortunately the usb 2.0 <-> 3.0 mismatch did not seem to fix the issue for me. To clarify, though, are you passing through physical ports or USB IDs? I am using vendor IDs for my devices, not the port...