[SOLVED] PCIe passthrough breaks display of memory usage

[deuko]

Hello community!

I am running a single node on Proxmox 8.1.4 and I successfully set up a Ubuntu Server 23.10 VM with GPU passthrough of my Intel Arc A750.
But when starting the VM I noticed something very strange. As soon as I added the PCI device to the VM and started it, it displays in the summary of the VM, that it nearly uses all the RAM. At first I thought I didn't allocate enough (initially I had 8 gigs configured), but after allocating 16 gigs it also immediately jumped which raised my suspicion.

I sshd into the machine and ran btop, which showed that the display of Proxmox GUI was wrong. As you can see in the screenshot it idles at around 630 MB which is more reasonable for a fresh Ubuntu Server install with just Jellyfin.

Any idea what causes this? What more info do you need?

 

[Azunai333]

If you use PCIe passthrough the complete RAM of the VM needs to be allocated at startup because the passthroughed devices need direct RAM mapping.
So even if your VM is not using it, PVE can not use it for anything else.

 

[deuko]

Ohh wow thanks for the fast reply. Sorry I did not know that, so I guess its solved then.

 

[rokyo401]

And there is currently (ever?) no way to stop this, right? As soon as you pass through anything PCI, it will make Proxmox "eat" all the RAM you assigned to that VM?

Does this have any negative effects besides the RAM not being available to the host and other VMs? Like on longevity of the RAM sticks or performance of host and other VMs?

 

[Azunai333]

And there is currently (ever?) no way to stop this, right?

No. An IOMMU-device needs DMA (Direct Memory Access) to the virtual machine RAM. This means it can modify/read the given RAM by itself, no CPU involved. This is why it has to be reserved at the startup of the VM.

Does this have any negative effects besides the RAM not being available to the host and other VMs? Like on longevity of the RAM sticks or performance of host and other VMs?

No, none.

 

[rokyo401]

No. An IOMMU-device needs DMA (Direct Memory Access) to the virtual machine RAM. This means it can modify/read the given RAM by itself, no CPU involved. This is why it has to be reserved at the startup of the VM.

Oh okay, but how does this work on a physical machine, then? There has to be some mechanism (in the OS or driver, maybe?) that prevents the DMA device from writing/reading to/from a memory region that is already in use by the OS or other devices or programs, right?