NVIDIA GPU Changing Slot ID

[Neil68]

I have a server running Proxmox that I set up a VM on to run Ollama. I went through the GPU passthrough configuration and everything was working great. This was running for almost a month. Then I decided to add another VM to the server running CodeProject.AI and also passthrough the GPU. I found that I can't run two VM's using the same GPU when I tried to start the CodeProject.AI server. I then shutdown the Ollama server so that I could use the GPU to work on the CodeProject.AI server. After running into issues with that, I shut it down and was using the Ollama server and noticed it was running slow. After some investigation I discovered that it was no longer using the GPU. In the hardware configuration for the VM it shows as having slot ID 42 but when I run the NVIDIA utility in the VM, it is showing as slot ID 01 and Off. I also tried shutting down the Ollama server and checking the CodeProject.AI server which is also showing slot ID 01. I have tried several things with the hardware configuration and checking the installation of the drivers/utilities for the GPU with no luck. I'm not sure what would be useful since I'm new to Proxmox, but here is my system info:

Dell PowerEdge R720
2 x Intel Xeon E5-2680v2@2.8 GHz
384 GB ECC RAM
500 GB SSD
NVIDIA Tesla P40 in PCIe slot 4
Proxmox VE 8.2.4
Ollama server is running PopOS
CodeProject.AI server is running Ubuntu 24.04.1 LTS

Any help or suggestions would be greatly appreciated.

 

[Pifouney]

nothing abnormal with the 01 ID in yours VM => You 've only passed trough One hardware, so he's only using the first ID slot.

Passing trough = > Dedicating the hardware defined for be only visible in the VM used for. 2 VMs cannot share in the same time the same consumer GPU.

For reset any kind of hardware cache, reboot your Hypervisor and test performance in Ollama. If good now, shutdown it. Then, start your another VM, and check if running well.
If not, compare cfg of your 2 VM. Maybe a bad VM type or bad BIOs type.
For passing trough, UEFI needed, q35-machine type too.

 

[Neil68]

nothing abnormal with the 01 ID in yours VM => You 've only passed trough One hardware, so he's only using the first ID slot.

Passing trough = > Dedicating the hardware defined for be only visible in the VM used for. 2 VMs cannot share in the same time the same consumer GPU.

For reset any kind of hardware cache, reboot your Hypervisor and test performance in Ollama. If good now, shutdown it. Then, start your another VM, and check if running well.
If not, compare cfg of your 2 VM. Maybe a bad VM type or bad BIOs type.
For passing trough, UEFI needed, q35-machine type too.

When it was working previously the slot ID was 42. Now that it has changed to 01 it doesn't work. I did discover that 2 VMs cannot share the same GPU. I'm OK with that. I just want to get the GPU working again. I tried rebooting the hypervisor like you suggested with no change. I am not using UEFI and it did work with BIOS. That might be something I could look into. I do have the machine type set to q35. Any other ideas are welcome.

 

[Pifouney]

i woul d suggest you to turn of secures boot in UEFI guest Bios, but if you're not in UEFI mode ....
Did you tried to passthrough "Mapped Device" option ?
Then with your card, you can use the GPU ressources with yours two VM at the same time. BUT, this only can work if using "Mapped Device", not 'raw device"

 

[Pifouney]

Did you tried to, in hardware section of your VM, to reallocate the GPU? it may be change og iommu group after update

 

[Neil68]

I did try reallocating the GPU in the hardware section of the VM with no change. When I have time I am going to build a new VM using UEFI and see if that works. I don't think I can change from BIOS to UEFI after the VM is built. Maybe I'll check into that too.