Maybe you are using older drivers in which NVidia actively breaks passthrough? Maybe their newer drivers don't support your GPU? Maybe you need to hide the virtualization from the drivers?
Maybe the device does not reset properly? Maybe the GPU is used during the boot of the computer (single GPU)? Maybe you need to patch the ROM of the GPU?
It's hard to tell without seeing the VM configuration file. Personally, I know very little about NVidia but maybe someone knows which of the above applies to your GPU.