Record PCI device usage of VMs

[mert98]

Hello Everyone;

I am running a Proxmox 7.1-7 server which contains 4 nodes and total of 9 GPUs. I have virtualized GPUs with no problem and each of them running smoothly on assigned VMs. However since our GPU capacity is limited and there is more than 9 VMs on server. I have to track gpu usage of each VM, so I can add/remove gpu between VMs for maximum efficiency.
I am asking, is there any way to daily/weekly record gpu usage of each VM ? So I can track GPU usage of each VM without chatting with each vm user

 

[Chris]

Hi,
what exactly do you mean by recording the GPU usage of each VM? Do you mean just track which GPU is passed trough to which VM, or do you mean see if the GPU attached to a given VM is actively used by the user of the VM?

In the first case you could get the information about which GPU is currently attached to which VM by writing a small script parsing the VM configuration files in /etc/pve/qemu-server.

In the latter case, I suppose you would have to install some kind of metric software within the VM.

 

[mert98]

Hello thanks for fast answer. I meant latter case, to see if the GPU attached to a given VM is actively used by the user or not. Is there any metric software you can suggest for this kind of task ? I am running Ubuntu server VMs. Or is there any way to do it by just using Proxmox instead of installing software to each VM ?

 

[Chris]

Hello thanks for fast answer. I meant latter case, to see if the GPU attached to a given VM is actively used by the user or not. Is there any metric software you can suggest for this kind of task ? I am running Ubuntu server VMs. Or is there any way to do it by just using Proxmox instead of installing software to each VM ?

I am afraid I cannot give you an out of the box solution here.
Just using the information from the Proxmox host is AFAIK not possible.

For GPU monitoring within the VM there are tools like nvidia-smi or intel-gpu-tools or radeontop available. To further monitor them from outside the VM, one would probably need to implement a custom exporter, (something like https://github.com/a0s/nvidia-smi-exporter).