I think I might have just gotten it!
I removed the input from the qm set command and configured it manually.
In my case the card to passthrough was PCI 01:00.0 and 01:00.1 (dual port)
So I added
hostpci0: 01:00.0,driver=vfio
hostpci1: 01:00.1,driver=vfio
Note: this was after manually unbinding...