Can't add multiple vGPUs to a VM

[gerog]

Hey everyone!
I tried to add two vGPU instances to the same VM, but it wouldn't start up. This is the error message I get:

kvm: -device vfio-pci,sysfsdev=/sys/bus/mdev/devices/00000000-0000-0000-0000-000000000102,id=hostpci0,bus=pci.0,addr=0x10: warning: vfio 00000000-0000-0000-0000-000000000102: Could not enable error recovery for the device
kvm: -device vfio-pci,sysfsdev=/sys/bus/mdev/devices/00000001-0000-0000-0000-000000000102,id=hostpci1,bus=pci.0,addr=0x11: vfio 00000001-0000-0000-0000-000000000102: error getting device from group 93: Input/output error
Verify all devices in group 93 are bound to vfio-<bus> or pci-stub and not already in use
waited 10 seconds for mediated device driver finishing clean up
actively clean up mediated device with UUID 00000001-0000-0000-0000-000000000102
waited 10 seconds for mediated device driver finishing clean up
actively clean up mediated device with UUID 00000000-0000-0000-0000-000000000102
TASK ERROR: start failed: QEMU exited with code 1

Is this even a thing, is it possible?
Thanks in advance.

 

[fireon]

How did you setup your vGPU? Did you used this tutorials?

https://pve.proxmox.com/wiki/PCI_Passthrough
https://pve.proxmox.com/wiki/NVIDIA_vGPU_on_Proxmox_VE

What GPU you would like to pass through?

 

[dcsapak]

Is this even a thing, is it possible?

depends a bit on the card, you should be able to see that with nvidia-smi:

# nvidia-smi vgpu -c -v
GPU 00000000:01:00.0                                                                                                                                                                                                                                                           
    vGPU Type ID                          : 0x291                                                                                                                                                                                                                              
        Name                              : NVIDIA RTXA5000-1B                                                                                                                                                                                                                 
        Class                             : NVS                                                                                                                                                                                                                                
        GPU Instance Profile ID           : N/A                                                                                                                                                                                                                                
        Max Instances                     : 24                                                                                                                                                                                                                                 
        Max Instances Per VM              : 1                                                                                                                                                                                                                                  
    ...
    vGPU Type ID                          : 0x293                                                                                                                                                                                                                              
        Name                              : NVIDIA RTXA5000-1Q                                                                                                                                                                                                                 
        Class                             : Quadro                                                                                                                                                                                                                             
        GPU Instance Profile ID           : N/A                                                                                                                                                                                                                                
        Max Instances                     : 24                                                                                                                                                                                                                                 
        Max Instances Per VM              : 16

so for example for the -1B variant it's not possible, for the -1Q variant it should be

 

[gerog]

How did you setup your vGPU? Did you used this tutorials?

https://pve.proxmox.com/wiki/PCI_Passthrough
https://pve.proxmox.com/wiki/NVIDIA_vGPU_on_Proxmox_VE

What GPU you would like to pass through?

Yes, I did use this.

depends a bit on the card, you should be able to see that with nvidia-smi:

# nvidia-smi vgpu -c -v
GPU 00000000:01:00.0                                                                                                                                                                                                                                                         
    vGPU Type ID                          : 0x291                                                                                                                                                                                                                            
        Name                              : NVIDIA RTXA5000-1B                                                                                                                                                                                                               
        Class                             : NVS                                                                                                                                                                                                                              
        GPU Instance Profile ID           : N/A                                                                                                                                                                                                                              
        Max Instances                     : 24                                                                                                                                                                                                                               
        Max Instances Per VM              : 1                                                                                                                                                                                                                                
    ...
    vGPU Type ID                          : 0x293                                                                                                                                                                                                                            
        Name                              : NVIDIA RTXA5000-1Q                                                                                                                                                                                                               
        Class                             : Quadro                                                                                                                                                                                                                           
        GPU Instance Profile ID           : N/A                                                                                                                                                                                                                              
        Max Instances                     : 24                                                                                                                                                                                                                               
        Max Instances Per VM              : 16

so for example for the -1B variant it's not possible, for the -1Q variant it should be

When I run this command, it tells me that the "Max Instances Per VM" is 1 for every instance. Just for clarification, I have two NVIDIA Tesla P40s. which seems to support vGPUs and it works perfectly with only one vGPU assigned to the VM.

 

[dcsapak]

When I run this command, it tells me that the "Max Instances Per VM" is 1 for every instance. Just for clarification, I have two NVIDIA Tesla P40s. which seems to support vGPUs and it works perfectly with only one vGPU assigned to the VM.

if the driver says that, it probably isn't supported for that gpu/driver, sorry