Has anyone figured out multiple physical GPU vGPU profiles to a single VM?

[zenowl77]

as the title states, i am trying to pass multiple vGPU profiles for multiple physical GPUs to the a single VM, it works fine for different VMs, but when i try to pass them to the same VM it just will not work and it also does not help that one GPU is a 8gb (Tesla P4) and the other is a 3GB (GTX 1060), so i am passing 7GB and 2GB profiles off to the VM.

currently i have an Intel A310 4GB passed to the VM, along with the Tesla P4 7GB profile, but i would like to add the GTX 1060 2GB profile on the same VM so that i can use the TeslaP4+GTX 1060 on multi GPU tasks that require the same type of graphics card and will not mix intel+nvidia.

The 1060 is also the proxmox systems primary GPU and seems to be using it for the VM rendering with VirGL GPU, LXC configurations, etc and is configured for mDev, so not sure i can just pass the whole device through without causing problems or re-configuring a lot

 

[zenowl77]

the best i can get working is passing the whole 1060 and vGPU profile for the P4 but then the drivers are incompatible and only one or the other will install inside windows, so i need both running vGPU profiles.

if i set them both to vGPU profiles and attempt to start the VM i get input/output (83) errors and it will only boot with one or the other.

swtpm_setup: Not overwriting existing state file.
kvm: -device vfio-pci,sysfsdev=/sys/bus/mdev/devices/00000002-0000-0000-0000-000000000127,id=hostpci2,bus=pci.0,addr=0x1b: warning: vfio 00000002-0000-0000-0000-000000000127: Could not enable error recovery for the device
kvm: -device vfio-pci,sysfsdev=/sys/bus/mdev/devices/00000003-0000-0000-0000-000000000127,id=hostpci3,bus=pci.0,addr=0x1c: vfio 00000003-0000-0000-0000-000000000127: error getting device from group 83: Input/output error
Verify all devices in group 83 are bound to vfio-<bus> or pci-stub and not already in use
stopping swtpm instance (pid 3755) due to QEMU startup error
waited 10 seconds for mediated device driver finishing clean up
actively clean up mediated device with UUID 00000003-0000-0000-0000-000000000127
waited 10 seconds for mediated device driver finishing clean up
actively clean up mediated device with UUID 00000002-0000-0000-0000-000000000127
TASK ERROR: start failed: QEMU exited with code 1

another fun part is i am trying to set two different profiles as the tesla has 8gb and the 1060 is the 3GB model so i want to maximize vram from both since i am using them for ai and GPGPU tasks.

i have these profiles set in profile_override.toml for the separate cards ( i have tried switching the number 156 with other profile numbers to test compatibility)

[profile.nvidia-55]
display_width = 3840
display_height = 2160
max_pixels = 8294400
cuda_enabled = 1
framebuffer = 0x1A0000000
framebuffer_reservation = 0x20000000 # 7GB

[profile.nvidia-156]
display_width = 3840
display_height = 2160
max_pixels = 8294400
cuda_enabled = 1
framebuffer = 0x74000000
framebuffer_reservation = 0xC000000 # 2GB

i am wondering if vGPU unlock can only handle one single card at a time? and if so how to i set it to unlock the 1060 and leave the P4....