sound like your gpu is sharing the same iommu group than your nic.
when you passthough a device, all devices from the iommu group are detached. If you have a good motherboard or a server, you should have 1 ommu group by device.
Maybe try to change your gpu physical slots if you have another pci express port free.