Multiple GPU system, would like to have 2 dedicated VMs with direct monitor and usb, need help.

[ZachH83]

The end game goal:
1 VM as a ubuntu server web server.
2 VM, each with a dedicated nvidia p400 plugged into a monitor and 2 usb ports running ubuntu desktop.

3 total VMS, 2 of which need gpu's passed through.

I am attempting to make 3 systems out of one. I've used all the hardware in a proxmox environment before, so I know its all good. I've also used both GPUS passed through to their own VM, which were used for video editing from a remote desktop, but never used with a monitor plugged into one of their ports.

I've had odd success when playing with this, and ended up getting a VM to display on the monitors, but that was short lived.

I've now gone through numerous walkthroughs on how to get this set up, and none of them are yield any success.

Does anyone know the proper way to go through this?
Also, is it normal for both gpus to have the same lspci -n -s result? the 23:00 and the 30:00 both return the same 2 10de:XXXX. I feel like the first time I had past them through the system for the old VMs, that I had to put 4 different results in the vfio.conf file? But, I could be wrong about that. Been messing with this for so long now, I feel like I can't tell which way is up.

 

[ZachH83]

Currently, I have both GPUs passed through to each VM. Both detected no problem.
Now, when I connect a cable to the gpu and the monitor, nothing happens. I have a monitor plugged into each card, but nothing is happening.
I can still console in no problem, but the cards are not detecting the monitor in the system.

 

[ZachH83]

While attempting to use the primary display option within the proxmox add pci hardware, I was not able to start the machine. It said that the vga was not allowed.

However, I edited the vfio.conf file that you edit when setting up the gpu passthrough, and changed the disable_vga=1 to =0.

I was then able to start the machine with the x-vga=on option (primary display).

Now, I have one machine on display, but the other one is getting an nvidia-modeset errior. So, I guess I will figure that one out next.