Okay, so apparently I needed to make the default GPU the intel integrated one, otherwise it wasn't fully letting go of the nvidia one.
But now, with the settings "balloon: 0" and "vfio_iommu_type1.allow_unsafe_interrupts=1 pci-stub.ids=<ids>" it just bluescreens.
It's the same both with and...