GPU VRAM sharing

[avladulescu]

Hello folks,


I would be interested to find out if any of you using vGPU in your VMs, managed to get the VRAM sharing working.

I tried scrubbing the forum and the Internet on this - if this is achievable or not, besides just allocating the usual nvidia specific profile sets for designated VRAM.



I've got a Tesla P4 (NVIDIA-SMI 535.161.05) hooked-up a Win11 VM with nvidia grid drivers installed and working, on a prox 7.4-19 node.

My VM configuration looks like below:

agent: 1,fstrim_cloned_disks=1
audio0: device=ich9-intel-hda,driver=none
bios: ovmf
boot: order=virtio0
cores: 8
cpu: host
cpuunits: 100
efidisk0: cstore00:881/base-881-disk-0.qcow2/9004/vm-9004-disk-0.qcow2,efitype=4m,pre-enrolled-keys=1,size=528K
hostpci0: 0000:08:00.0,mdev=nvidia-54
machine: pc-q35-7.2
memory: 16384
name: W11
net0: virtio=C6:E0:2A:3A:BD:98,bridge=vmbr0,firewall=1,tag=3202
numa: 1
onboot: 1
ostype: win11
scsihw: virtio-scsi-pci
smbios1: uuid=6323b609-4186-41c3-b0a0-37c5156c211a
sockets: 1
vga: virtio-gl,memory=512
virtio0: cstore00:881/base-881-disk-1.qcow2/9004/vm-9004-disk-1.qcow2,discard=on,iothread=1,size=250G
vmgenid: e9802124-727e-42a4-80bf-a44beb7008f3

Note on ( vga: virtio-gl,memory=512 ) - not quite sure if memory declaration here does have an effect or not on the system. I could see it capped at 512M, but inside the Win VM, in device manager the size doesn't get reported on the primary Microsoft driver.


Inside the VM, under GPU (task manager) the status looks like the following:



I see the dedicate VRAM portion being used depending on the RDP and started apps, but I can't figure it out if Shared GPU VRAM has a special meaning. Is there anything like memory ballooning or KSM implemented for VRAM allocation besides getting a bigger card?

Syslog reports "Guest attempted to allocate channel above its max channel limit 0xa2" and Nvidia docs ( https://docs.nvidia.com/vgpu/sizing/virtual-workstation/latest/best-practices.html ) : "When the channels allocated to a vGPU are exhausted, and the guest VM fails to allocate a channel, the following errors are reported on the hypervisor host or in an NVIDIA bug report"

The point would be to split and map the GPU on multiple Win VMs and have a baseline of (for eg. 512MB of VRAM allocated in each VM), whereas the unused GPU VRAM memory, to be dinamically allocated under shared GPU VRAM between the VMs (ofc, considering the VRAM on the phy GPU card usage is not used at 100%).


Any knowledgeble feedback is appreciated.

Thank you,
Alex

 

[dcsapak]

i don't think that the vram for vgpu can be shared in any way, (at least not as far as i know), so when you assign a specific vgpu model, it only has the predetermined amount of vram
I think what windows means here with 'shared gpu memory' is a part of the normal ram that the gpu might use if it has not enough (like 'swapping' but for vram instead of memory)

also note that there are no real drivers for virtio-gl for windows at the moment, so that setting is not really necessary or helpful (using the default should be enough)

 

[avladulescu]

Hi Dominik,

Thanks for the reply.

After some digging around, I could understand that the shared portion of VRAM is basically from the RAM itself, as some windows processes need also the GPU context switching in the WDDM driver, therefore that's not usable directly as video RAM memory nor can it be resized or changed.

Regarding the Virtio GL I understand it's more like holly water - it doesn't help nor it does bad on the system.

My personal challenge would be only to render the RDP connections in the GPU, without mixing other apps that use WDDM, but so far I could see that this is not changeable or separated in the windows user context.

As a side note, I could find some forums over the internet about users complaining on NVIDIA about vRAM sharing and after some reading I understand that AMD and Intel IGP has addressed that in their drivers, but NVIDIA no.

Regards,
Alex