So this actually turned out to be an issue with the NIC itself. The hardware for the second port seems to hang once it is passed through to a guest OS. Even Proxmox (after a warm boot) showed the same error when initializing the second port on the NIC. The only way to get the second port...