vGPU - Nvidia vm - vm does not start

[deepcloud]

Hi

I have a 48GB A40 GPU running Proxmox 8.2 (pve-manager/8.2.4/faa83925c9641325 (running kernel: 6.5.13-3-pve)) with nvidia driver ver. 535.161.05

I am running 7 vm with vgpu successfully (4GB Profile)

I should be able to run 12 VM with the given video memory I have.

But not able to start the 8th VM. upon starting I get the below message
------------------------------------------------------------------------------------------------------------------
/dev/rbd40
kvm: -device vfio-pci,host=0000:81:01.3,id=hostpci0,bus=pci.0,addr=0x10: vfio 0000:81:01.3: failed to setup container for group 62: Failed to set group container: Device or resource busy
start failed: QEMU exited with code 1
-----------------------------------------------------------------------------------------------------------------

any ideas

Thanks